A blockchain data chaining method and system based on a message queue

Abstract

The application discloses a blockchain data chaining method and system based on a message queue, and belongs to the technical field of blockchain data processing, and comprises the following steps: writing a request carrying a global business idempotent identifier and a calling party account identifier into a message queue; performing consistent hash routing based on the account identifier, and scheduling the same account request to a fixed consumption thread; when the thread is initialized or rebalanced, synchronously acquiring a ledger confirmation serial number, a memory pool maximum serial number and a cache high water level, taking the maximum value to rebuild a serial number sliding window, and pre-allocating an incremental serial number for submission and chaining; periodically detecting a suspended transaction queue, identifying a queue head blocking cavity, and sending a placeholder transaction to dredge; and a contract on the chain verifies the business idempotent identifier based on a time epoch hierarchical rolling state mapping tree, and periodically prunes historical states. While introducing a message queue for asynchronous peak clipping, the application can effectively solve the problems of concurrent serial number conflicts, rebalancing stampede, queue head blocking and on-chain state explosion.

Description

Technical Field

[0001] This invention relates to the field of blockchain data processing technology, and in particular to a method and system for uploading blockchain data based on message queues. Background Technology

[0002] With the deepening application of blockchain technology across various industries, the scale of data interaction between business systems and underlying blockchain nodes is constantly surging. In actual production, when facing peak business periods, massive data upload requests often arrive in a concentrated manner. Due to the limited capacity and consensus throughput of the blockchain's underlying node's transaction mempool, the concentrated arrival of massive requests can cause nodes to directly discard transactions.

[0003] To address this, the industry commonly introduces message queues (MQ) at the access layer to smooth out and fill in the peaks of sudden upstream traffic. Then, backend consumers asynchronously construct blockchain transactions and submit them to the chain nodes to reduce the blocking of business request threads and improve overall throughput.

[0004] However, while the introduction of message queues (MQ) alleviated throughput pressure, it introduced new distributed challenges because transactions issued by a single account rely on strictly incremental transaction sequence numbers for sorting and validity verification. Specifically, since MQ allows multiple consumer threads to pull messages in parallel, when multiple on-chain requests from the same account are processed concurrently by different threads, each thread reads the current sequence number returned by the node from its local cache, which can easily lead to duplicate allocation, skipped allocation, or out-of-order submissions, thus triggering transaction rejection or subsequent transaction backlog. To address these concurrency conflicts, a common approach is to use a consistent hashing algorithm to route requests from the same account to a fixed consumer thread. Under normal circumstances, this approach can effectively avoid concurrent Nonce contention. However, when the consumer cluster undergoes scaling up or down, node restarts, or network fluctuations trigger partition rebalancing, the newly taken-over thread loses its memory context and cannot perceive the previously allocated but unconfirmed Nonce status. Its newly allocated sequence number is very likely to overlap with transactions already queued in the node's memory pool. Furthermore, in the event of network jitter or packet loss, if a transaction with a smaller sequence number fails to reach a blockchain node, the node will place subsequent legitimate transactions that have already arrived into a pending queue and wait indefinitely for the preceding transactions, causing congestion at the head of the queue for that account and completely paralyzing the channel for subsequent legitimate transactions. Further, in the aforementioned asynchronous commit mode, various preceding scenarios (concurrency conflicts, rebalancing collapse, network packet loss) will trigger frequent transaction retries. To prevent retries from causing duplicate execution of business logic, smart contracts need to maintain a set of anti-replay identifiers for processed business transactions. However, with hundreds of millions of transactions, if permanent cumulative storage is used, this anti-replay record will expand indefinitely, leading to an exponential increase in addressing costs and ultimately crippling the blockchain's read performance.

[0005] Existing mechanisms are insufficient to address the concurrency conflicts, state loss, head-of-line blocking, and state explosion issues arising from the combination of Nonce continuity constraints and the high cost of on-chain storage in an asynchronous buffer architecture. Summary of the Invention

[0006] The purpose of this invention is to provide a blockchain data uploading method and system based on message queues, so as to solve the problems of sequence number conflict, consumer rebalancing, queue head blocking and on-chain state bloat caused by the coupling of message queue consumption model and blockchain account sequence number model in the asynchronous concurrent uploading scenario in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for uploading blockchain data based on a message queue, comprising: Receive data upload requests carrying a global business idempotency identifier and a caller account identifier and persist them to the message queue; Consistent hashing is performed based on the caller's account identifier to schedule requests from the same account to a fixed asynchronous consumption thread, thereby establishing a serial processing relationship between the account and the thread on the application side and reducing the risk of sequence number disorder caused by concurrent consumption. When the asynchronous consumption thread is initialized and / or when partition rebalancing, thread migration or fault recovery occurs in the message queue, it simultaneously pulls the blockchain ledger confirmation sequence number, the maximum unpacked sequence number in the node memory pool and the distributed cache high watermark, and takes the maximum value of the three to reconstruct the local sequence number sliding window. Pre-allocate strictly incremental transaction sequence numbers from the reconstructed sequence number sliding window, generate transactions carrying a global business idempotency flag and submit them to the blockchain node, while simultaneously writing them to the cache high watermark; Periodically detect the pending transaction queues of each account, identify the blockage gaps at the head of the queue caused by missing preceding sequence numbers, and proactively construct and send placeholder transactions with missing sequence numbers to clear the blockage. The on-chain smart contract performs an existence check on the global business idempotency identifier in a rolling state mapping tree based on time epochs. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the execution of repeated business is blocked. Furthermore, when the rolling state mapping tree reaches the preset number of epochs to be retained, the historical mappings of the expired epochs are periodically pruned and destroyed.

[0008] Furthermore, the step of reconstructing the local sequence number sliding window by taking the maximum value of the three includes: Query the maximum transaction sequence number of the most recently confirmed on-chain transaction of the calling account mentioned in the blockchain ledger, and denote it as N1; Query the maximum transaction sequence number of the transactions of the caller account that are pending packaging, suspended or propagated in the blockchain node memory pool, and denote it as N2; Read the high-water mark of the transaction that the caller account has been allocated or committed, recorded in the distributed cache, and denote it as N3; Pick As the baseline value for reconstruction, and according to [N] base+1 , N base+W Initialize the serial number sliding window in local memory, where W is the preset window size.

[0009] Each sequence number in the sequence number sliding window has a status identifier, which includes at least: FREE (assignable), RESERVED (pre-assigned but not yet sent), SUBMITTED (sent but awaiting confirmation), CONFIRMED (confirmed), and ABNORMAL (abnormal and awaiting compensation).

[0010] The aforementioned tripartite reconciliation mechanism can simultaneously detect the following three types of state differences: a transaction has been confirmed on the blockchain but the cache has not yet been updated; a transaction has been received by a node and entered the memory pool's pending queue but the ledger has not yet been confirmed; a transaction has been allocated by the application layer and written to the cache but failed to be broadcast to the node due to network anomalies. By taking the maximum value of these three as the window reconstruction benchmark, it is possible to avoid reusing already occupied sequence numbers after recovery, while also reducing unnecessary large-span number jumps.

[0011] Furthermore, the sliding window reconstruction is triggered in any of the following scenarios: the asynchronous consumer thread takes over the caller account for the first time after starting; the message queue undergoes partition rebalancing, causing the caller account to migrate from the current asynchronous consumer thread to another asynchronous consumer thread; the asynchronous consumer thread detects that the sequence number allocation status in local memory is lost or the cache status is rolled back; or the asynchronous consumer thread experiences a crash recovery or a network interruption recovery.

[0012] Furthermore, the active construction of placeholder transactions with missing sequence numbers for unblocking includes: The list of pending transactions of the caller's account in the blockchain node is retrieved at a preset scanning cycle; Identify head-of-line blocking holes based on gaps in consecutive transaction sequence numbers in the pending transaction list; When the minimum pending transaction sequence number is detected to be greater than the ledger confirmation sequence number plus 1, it is determined that there is a queue head blockage caused by the loss of the preceding transaction sequence number; A placeholder transaction is generated for the missing sequence number corresponding to the blockage hole at the head of the queue, and a compensation submission is initiated to the blockchain node.

[0013] The placeholder transaction is an empty transaction that does not trigger actual business execution. Its transaction payload is either empty or contains a preset debugging identifier field, and the transaction recipient address is a preset empty operation contract address. After the placeholder transaction is uploaded to the blockchain, the off-chain index service listens for new block events or periodically pulls block data to obtain the transaction payload that has been uploaded to the blockchain. If the preset debugging identifier field is detected, the transaction is marked with a compensation placeholder tag. When performing business queries, statistics, or reconciliation outputs, the empty transaction is removed from the valid business dataset, and a compensation log is retained for operation and maintenance traceability.

[0014] Furthermore, in the rolling state mapping tree based on time epochs, time epochs are divided in any of the following ways: divided by block height intervals, with a time epoch formed after a preset number of blocks; divided by on-chain timestamp intervals, with a time epoch formed after a preset duration; or divided jointly by business batch cycle and block height.

[0015] The addressing depth of the rolling state mapping tree remains fixed, and the current epoch state mapping and the historical epoch state mapping are organized using a isomorphic trie. When verifying the business idempotency identifier, the on-chain smart contract performs a limited-level search in the order of current epoch priority and historical epoch backtracking.

[0016] When the cumulative number of epochs reaches a preset threshold N, an asynchronous pruning task is triggered. The state mappings corresponding to the most recent M epochs are retained, and the root nodes of the historical dictionary tree earlier than the most recent M epochs are unbound, deleted, or made unreachable to release the on-chain state and block the repeated access paths of the idempotent flag of overdue business.

[0017] Furthermore, the method is applied to high-frequency asynchronous concurrent data on-chain scenarios, which include one or more of the following: transaction flow notarization, order event on-chain, settlement result on-chain, log summary on-chain, or device telemetry data on-chain.

[0018] This invention also provides a blockchain data uploading system based on message queues, comprising: The request access module is used to receive data uplink requests carrying a global business idempotency identifier and a caller account identifier, and to persist the data uplink requests to a message queue. The hash grouping scheduling module is used to perform consistent hash routing based on the caller account identifier, and schedule data uploading requests from the same caller account to a fixed asynchronous consumption thread; The three-party reconciliation and window reconstruction module is used to synchronously obtain the confirmed sequence number in the blockchain ledger, the maximum sequence number to be packaged in the node memory pool, and the high-water mark in the distributed cache when the asynchronous consumption thread is initialized and / or when partition rebalancing, thread migration, or fault recovery occurs in the message queue, and reconstruct the local sequence number sliding window with the maximum value. The sequence number pre-allocation and on-chain submission module is used to pre-allocate strictly increasing transaction sequence numbers from the sliding window, construct blockchain transactions containing global business idempotent identifiers, and update the high-water mark in the distributed cache. The thread scheduling module is used to control the migration of the caller account from the current asynchronous consumer thread to the target asynchronous consumer thread during the scaling up or down of the asynchronous consumer thread cluster or partition rebalancing. During the migration process, it controls the old asynchronous consumer thread to perform a drain operation, and controls the new asynchronous consumer thread to take over and trigger the third-party reconciliation and reconstruction after the drain is completed. The timed compensation module is used to periodically detect missing sequence number gaps in the suspended transaction queue and construct placeholder transactions to clear them. The on-chain anti-replay execution module is used to perform existence checks on the global business idempotent identifier in the rolling state mapping tree based on time epoch hierarchy. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the repeated business execution is blocked, and the historical mapping of the expired epoch is periodically pruned and destroyed.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a blockchain data uploading method and system based on message queues. By employing consistent hashing, requests from the same account are routed to a fixed asynchronous consumption thread, avoiding the sequence number contention and out-of-order issues caused by multiple threads concurrently competing for the same account's Nonce resource. This not only improves the accuracy of Nonce allocation in high-concurrency scenarios but also reduces the consumption of system resources due to numerous transaction retries caused by Invalid Nonce errors.

[0020] This invention enables asynchronous consumer threads to synchronously pull the blockchain ledger confirmation sequence number, the maximum sequence number to be packaged in the memory pool, and the high-water mark of the distributed cache for three-way reconciliation during initialization or when encountering partition rebalancing. Based on the reconciliation results, a local sequence number sliding window is reconstructed, allowing consumers to accurately perceive the current account's Nonce status after a crash recovery or cluster scaling. This helps improve the continuity of on-chain services in a distributed environment and avoids sequence number overlap and collapse caused by context loss.

[0021] This invention periodically probes the transaction queues of blockchain nodes through a timed compensation module to identify head-of-queue blocking gaps caused by missing preceding sequence numbers. The sequence number sliding window reconstructed by the third-party reconciliation mechanism provides an accurate baseline for the timed compensation module's gap detection, avoiding misjudgments or omissions of gaps during window sliding. Based on this, the timed compensation module periodically identifies head-of-queue blocking gaps, proactively constructs and submits empty transactions with missing sequence numbers to activate blocked subsequent legitimate transactions. This eliminates the blocking impact of network packet loss and other anomalies on the on-chain channel of a single account, further improving the system's availability in weak network environments.

[0022] This invention constructs a rolling state mapping tree based on time epochs in smart contracts, performs existence verification on global business idempotency identifiers, and periodically prunes and destroys historical dictionary trees that exceed preset expiration epochs. By periodically pruning and destroying the state mappings of expired epochs, it achieves bounded storage of replay-preventable state records, while limiting the addressing depth of the state mapping tree to a controllable range.

[0023] This invention also provides a distributed cache for persistently recording the current high-water mark sequence number of each account, enabling consumers to quickly restore the Nonce allocation state at any time by querying the cache. This improves the reliability of the entire system when dealing with operational operations such as consumer cluster restarts, scaling up, or scaling down. Furthermore, by comprehensively judging the states of the cache, ledger, and memory pool, the monotonicity and continuity of sequence number allocation are effectively guaranteed, providing an accurate judgment benchmark for the hole detection module. The three-layer collaboration of cache high-water mark, ledger state, and memory pool state jointly constructs a data on-chain mechanism that combines high throughput and strong consistency under a high-concurrency asynchronous buffer architecture. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of a blockchain data uploading method based on message queues according to the present invention; Figure 2 This is a schematic diagram of the on-chain request processing flow in this invention; Figure 3 This is a schematic diagram of the three-party dynamic reconciliation mechanism in this invention; Figure 4 This is a flowchart of the serial number pre-allocation and transaction submission process in this invention; Figure 5 This is a schematic diagram of the process in this invention where the timing compensation module identifies the blockage and void at the head of the queue and sends a placeholder transaction to fill and clear the gap. Figure 6 This is a schematic diagram of the structure of the present invention, which constructs a rolling state mapping tree based on time epochs and performs existence verification and history pruning. Figure 7This is a schematic diagram of the overall system architecture of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. All equivalent modifications or substitutions made based on the concept of the present invention should be covered within the scope of protection of the present invention. Unless otherwise expressly defined, the transaction sequence number in the following description refers to a monotonically increasing sequence number used in the account model blockchain system to characterize the order in which transactions are sent from the same initiating account, and may also correspond to the Nonce in the prior art. To avoid confusion in terminology, the transaction sequence number will be used uniformly below.

[0026] To facilitate a precise understanding of the technical solutions of this invention by those skilled in the art, the core terms used in this specification are defined as follows: High-water mark: In a distributed cache or equivalent persistent state component, the highest allocated or committed transaction sequence number recorded for a specific caller account. The high-water mark does not necessarily indicate that all transactions before the corresponding sequence number have been confirmed on-chain; rather, it indicates that the application, from the perspective of sequence number allocation, has considered that sequence number and the range below it as a non-reallocatable boundary. The high-water mark is used to quickly restore the allocation progress of the account during consumer restarts, migrations, rebalancing, or fault recovery, and participates in three-way reconciliation along with the ledger confirmation status and node mempool status.

[0027] Sequence number sliding window: A contiguous, finite-capacity range of candidate transaction sequence numbers maintained in the local memory of a caller account. This range is typically represented as [N...]. base+1 N base+W ], where N base The window is reconstructed using a baseline value, where W represents the window capacity. Each transaction sequence number within the window carries a status identifier, including at least: FREE (assignable), RESERVED (pre-assigned but not yet sent), SUBMITTED (sent but awaiting confirmation), CONFIRMED (confirmed), and ABNORMAL (abnormal and awaiting compensation). This sliding window is used on the application side to implement strictly incremental sequence number pre-assignment, status tracking, anomaly compensation, and window scrolling expansion.

[0028] Rolling State Map Tree: A state mapping structure organized hierarchically by time epoch for on-chain smart contracts to store and verify global business idempotency identifiers. Preferably, the structure is implemented at the underlying level using a Merkle-like trie corresponding to the blockchain platform's state database, such as the Merkle Patricia Trie (MPT) or other equivalent verifiable key-value trees. Each time epoch corresponds to a subtree of the current epoch state or a set of logical mapping root references, and different epochs are organized through the top-level EpochState index. The rolling state map tree is used to perform business idempotency verification within a finite time window and to suppress the infinite growth of on-chain state by pruning expired epoch states.

[0029] Time Epoch: A time segment unit used to divide the retention period of idempotent states in a business process. The time epoch can be divided according to block height range, on-chain timestamp range, or a combination of business batch period and block height. For example, an epoch can be defined as forming after a preset number of blocks, or after a preset duration (e.g., 1 hour, 1 day). The purpose of the time epoch is to provide a bounded retention window for on-chain replay protection states, ensuring that online replay protection verification is limited to the most recent M epochs, while earlier historical epochs can be pruned or moved to the archive path.

[0030] Example 1 This embodiment uses a high-frequency trading transaction log on-chain scenario as an example. Assume a fintech platform needs to write business data such as matching results, account asset change events, or clearing instruction summaries into a consortium blockchain in real time to meet requirements for auditing, multi-party verification, and regulatory reporting. In this application scenario, the peak request volume at the business entry point can reach tens of thousands of requests per second. To avoid long-term blocking caused by the front-end interface directly waiting for blockchain transaction receipts, the platform introduces a message queue (MQ) between the business system and the blockchain nodes to receive on-chain tasks. The consumer-side thread asynchronously retrieves tasks from the message queue and submits them to the blockchain.

[0031] In the asynchronous submission mode described above, if multiple transaction streams belong to the same calling account (i.e., the AccountId is the same), these requests must ultimately be written to the blockchain node with strictly increasing transaction sequence numbers. Blockchain systems using an account model require each transaction submitted by the same initiating account to carry a monotonically increasing transaction sequence number for node sorting and validity verification. If the asynchronous consumption thread fails to correctly maintain the continuous transaction sequence number of the account during high concurrency, rebalancing, or fault recovery, a large number of transactions will be rejected, suspended, or blocked by the node.

[0032] To address the aforementioned technical problems, this invention provides a method for uploading blockchain data based on message queues. For example... Figure 1 As shown, the method includes the following steps: Receive data upload requests carrying a global business idempotency identifier and a caller account identifier, and persist them to the message queue; Consistent hash routing is performed based on the caller's account identifier, and data upload requests from the same caller account are scheduled to a fixed asynchronous consumption thread to establish a serial processing relationship between account and thread on the application side, thereby reducing the risk of transaction sequence number disorder caused by concurrent consumption. During initialization and / or when partition rebalancing, crash recovery, network interruption recovery or local state loss occurs in the message queue, the asynchronous consumption thread synchronously pulls the confirmed maximum transaction sequence number in the blockchain ledger, the maximum transaction sequence number to be packaged in the node memory pool, and the high-water mark in the distributed cache, and takes the maximum value of the three to reconstruct the local sequence number sliding window. The asynchronous consumption thread pre-allocates strictly increasing transaction sequence numbers from the reconstructed sequence number sliding window, generates blockchain transactions carrying a global business idempotency flag, and submits them to the blockchain node, while updating the high-water mark in the distributed cache. The timed compensation module periodically probes the pending transaction queues of each account, identifies queue head blockages caused by missing preceding transaction sequence numbers, and proactively constructs placeholder transactions with missing transaction sequence numbers to send to the blockchain node in order to clear the blocked subsequent transactions. The on-chain smart contract extracts the global business idempotency identifier and performs an existence check in the rolling state mapping tree based on time epochs. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the execution of repeated business is blocked. Furthermore, when the rolling state mapping tree reaches the preset number of epochs to be retained, the historical mappings of the expired epochs are periodically pruned and destroyed.

[0033] The following combination Figures 2 to 6 The specific implementation methods for each step are described in detail.

[0034] like Figure 2 As shown, the data on-chain request processing flow provided in this embodiment of the invention includes three stages: request reception and verification, message persistent writing, and account-level consistent hash routing.

[0035] The business gateway receives data upload requests from upstream systems, and each request includes at least the following data fields: The global business idempotent identifier BizId is used to uniquely identify the same business action; The caller account identifier, AccountId, is used to determine the initiating account corresponding to the request in the underlying blockchain; The business payload is used to carry the transaction summary, evidence summary, or business fields to be uploaded to the blockchain; Create a timestamp Ts to record the time the request was generated; The Business Type (Type) indicates the business category to which the request belongs, such as transaction recording on the blockchain, asset change recording on the blockchain, or liquidation instruction recording on the blockchain. The optional Sign field is used by the recipient to verify the legitimacy of the request source and the integrity of the data.

[0036] Upon receiving the request, basic verification is first performed on the request. The basic verification includes at least format verification (checking whether each field conforms to the preset data type and length requirements), required field verification (confirming that fields such as BizId, AccountId, and Payload are not empty), and signature validity verification (verifying the correctness of the Sign field using a pre-configured public key).

[0037] After successful verification, the business gateway persists the request to the message queue. Preferably, the persistence process adopts an acknowledged write mode, that is, after the producer sends the message, it synchronously waits for the message queue server (Broker) to return an acknowledgment (ACK) to ensure that the request is recoverable on the message queue server before the consumer actually processes it, avoiding message loss due to Broker downtime.

[0038] After message persistence is completed, this embodiment performs secondary scheduling on the messages pulled from the message queue on the consumer side. Unlike conventional message middleware that distributes messages according to random or batch rules to improve throughput, this embodiment adopts an account-level consistent hash routing strategy based on AccountId.

[0039] Consistent hashing is a distributed addressing algorithm. Its basic principle is to map account identifiers to a ring-shaped space using a hash function, and then map asynchronous consumer threads to the same ring-shaped space. The target thread corresponding to an account is determined by searching clockwise. A key advantage of this algorithm is that when the asynchronous consumer thread pool expands or shrinks, only the local mapping relationships on the ring need to be adjusted, rather than being completely remapped.

[0040] Specifically, the system pre-maintains a pool of asynchronous consumer threads, denoted as the thread set. Where k is a positive integer representing the total number of threads. The specific execution steps of the consistent hash routing are as follows: Perform a hash calculation on the AccountId field in the message to obtain a hash value; The hash value is mapped to a consistent hash ring, and combined with the current thread slot (the position of the thread on the hash ring) distribution on the ring, a clockwise search is performed to find the thread slot that is closest to the hash value. The target thread corresponding to the thread slot is determined as the message processing thread, and all historical and subsequent messages with the same AccountId are continuously dispatched to the target thread for processing.

[0041] Through the aforementioned consistent hashing routing, this embodiment establishes a strict constraint on the application side that "only one thread is responsible for sequence number allocation for each account at any given time." In other words, messages belonging to different AccountIds can be processed concurrently by different threads, thereby achieving horizontal scaling; however, messages belonging to the same AccountId are always consumed serially by the same thread, thus avoiding problems such as duplicate allocation, skipped allocation, or out-of-order submission of transaction sequence numbers caused by multiple threads simultaneously reading the current transaction sequence number of the same account and generating transactions independently.

[0042] Furthermore, to reduce the jitter in account-thread mapping relationships caused by changes in the number of message queue partitions or the scaling up or down of consumer instances, the consistent hash routing in this embodiment preferably adopts a minimum migration amount implementation. The minimum migration amount means that when the asynchronous consumer thread pool is expanded (adding threads) or scaled down (removing threads), the slot distribution on the consistent hash ring only undergoes local adjustments, with only a small number of AccountId mapping targets changing, while the majority of AccountIds continue to be processed by the original threads. This minimizes the state reconstruction overhead and recovery time triggered by changes in mapping relationships.

[0043] For example, in a specific case, suppose the asynchronous consumer thread pool initially contains 5 threads and is expanded to 8 threads. After implementing consistent hashing with minimal migration, only about 3 / 8 of the AccountIds undergo mapping changes, while the remaining about 5 / 8 of the AccountIds retain their original mapping relationships, thus reducing the recovery cost of the system during scaling up or down.

[0044] When an asynchronous consumer thread takes over an AccountId for the first time, or takes over the AccountId again after a consumer group rebalancing, asynchronous consumer thread restart, thread migration, or network interruption recovery, it cannot directly rely on the old sequence number state in local memory to continue allocating sequence numbers. This is because the local state may have expired (e.g., the account had transactions processed by other threads before takeover and the transaction sequence number was advanced), been lost (e.g., the original thread crashed, causing the memory data to be cleared), or be inconsistent with the node memory pool state (e.g., some transactions have entered the node memory pool but have not yet been confirmed, and the application layer is unaware of this).

[0045] To address this, the present invention employs a three-party dynamic reconciliation mechanism combining ledger, memory pool, and cache. This three-party reconciliation refers to simultaneously obtaining the current transaction sequence number status of the account from three data sources: the blockchain ledger, the node transaction memory pool, and the distributed cache, and determining the true and valid current sequence number benchmark by taking the maximum value. This mechanism is specifically designed for deep recovery needs in scenarios such as asynchronous consumer migration and crash recovery.

[0046] The process of the three-party dynamic reconciliation mechanism is as follows: Figure 3 As shown, for any AccountId to be restored, the asynchronous consumer thread synchronously obtains the following three types of status sources: Ledger Confirmation Sequence Number N1: This value is obtained through the state query interface provided by the blockchain node, representing the largest sequence number of the most recently confirmed transactions on the blockchain for this account. This value reflects the state bound that has been solidified in the ledger; all sequence numbers less than or equal to this value have been confirmed on the blockchain and will not change further.

[0047] The maximum pending sequence number N2 in the memory pool is obtained through the node's suspended transaction query interface. This retrieves a list of pending or suspended transactions for this account that have not yet been confirmed but have been received by the node in the transaction memory pool (an internal node storage area for received but not yet packaged into blocks), and extracts the maximum sequence number from this list. This value reflects the status of a sequence number that has actually been used even though it hasn't been confirmed on the blockchain. Specifically, some sequence numbers less than or equal to this value may already be queued in the memory pool, and asynchronous consumption threads should not reassign these sequence numbers.

[0048] Distributed Cache Highwater Marker N3: After each successful transaction request is sent to a blockchain node, the application layer writes the corresponding sequence number to the distributed cache (such as a Redis cluster, Tair, or other key-value storage system) as a highwater mark for that account. This value reflects the upper bound of the most recently allocated or committed value on the application side.

[0049] Subsequently, the system calculates the baseline sequence number using the following formula:

[0050] Based on this, a sliding window of serial numbers is reconstructed in local memory. The sliding window refers to a continuous range of transaction serial numbers, within which serial numbers can be pre-allocated for use by the current thread. For example, if the window size W = 100, the reconstructed range is... Each serial number in this window has a defined status; example statuses include: FREE: This serial number has not been assigned and is available for collection. RESERVED (Pre-assigned): This sequence number has been claimed and pre-assigned by a thread, but has not yet been sent to a blockchain node; SUBMITTED: The transaction corresponding to this sequence number has been sent to the blockchain node and is awaiting confirmation; CONFIRMED: The transaction corresponding to this sequence number has been confirmed on-chain and has been fixed in the ledger; ABNORMAL (Abnormal): The transaction corresponding to this sequence number has not been confirmed for an extended period of time, requiring intervention from the compensation module.

[0051] Preferably, the window size W is a configurable parameter, which can be dynamically adjusted by those skilled in the art based on the account's historical transaction frequency and system memory resources. For example, W=128 or 256 can be configured for high-frequency trading accounts, and W=32 can be configured for low-frequency accounts, in order to achieve a balance between allocation efficiency and memory usage.

[0052] This embodiment uses a triple-source reconciliation mechanism to simultaneously detect the following differences. The table below summarizes these differences from three dimensions: the observed phenomena, the numerical relationships, and the effect of taking the maximum value.

[0053] When the three data sources are equal or there is no absolute leader, any data source can be used as a benchmark, and taking the maximum value will also yield the correct result. Taking the maximum value of the three can cover all the above-mentioned inconsistencies, ensuring that the benchmark value is not lower than the other data sources even when any data source is leading, thereby avoiding sequence number rollback or duplicate allocation.

[0054] In asynchronous consumption scenarios, when an asynchronous consumption thread resumes, some solutions only query the ledger confirmation sequence number (i.e., only obtain the maximum sequence number of transactions already on-chain), while others rely on historical allocation states in the local cache for recovery. The former cannot detect queued but unconfirmed transactions in the node's memory pool, while the latter cannot accurately recover when the cache is lost or delayed. This embodiment, by simultaneously pulling three types of state sources and taking the maximum value, can accurately recover the current sequence number state even when any one of the ledger, memory pool, or cache is in the lead position, without needing to preset the order of the three, thus exhibiting stronger fault tolerance and adaptability.

[0055] For example, if an account's ledger confirmation reaches 105, the node's maximum sequence number in the pending queue is 108, and the cache high-water mark is 107, it means that although 108 has not yet been confirmed, it has already been occupied by the node. Therefore, the starting sequence number after recovery should begin from 109, not 106 or 108. If recovery is based solely on the ledger value 105, sequences 106 to 108 may be sent repeatedly, causing the node to reject transactions due to sequence number conflicts. If recovery is based solely on the cached value 107, sequence number 108, which has already entered the memory pool, may be ignored, causing the newly allocated sequence number 108 to collide with the existing sequence number 108 in the memory pool. Therefore, this three-party reconciliation logic is the key mechanism of this invention for maintaining sequence number continuity in consumer migration and fault recovery scenarios.

[0056] Serial number pre-allocation and transaction submission process as follows Figure 4 As shown, it includes preprocessing before transaction construction, serial number acquisition and status update, transaction encapsulation and submission, post-submission status tracking, and window scrolling expansion.

[0057] After the asynchronous consumer thread retrieves a message from the message queue, it first performs preprocessing operations before transaction construction. This preprocessing includes at least: Verify message integrity (e.g., check if all fields of the message are complete, and whether there is any data truncation or corruption). Parse the business payload (extract the business data to be uploaded to the blockchain from the Payload field); Generate on-chain call parameters based on the business type (for example, construct different smart contract method call parameters based on different Type fields). Calculate transaction summaries (e.g., perform hash calculations on key business fields as a basis for transaction verifiability); Assemble metadata fields (encapsulate information such as BizId, AccountId summary, Type, Ts into the transaction payload).

[0058] After preprocessing, the thread checks if there is a sequence number in the local sliding window with a state of FREE: if there is no sequence number in the FREE state, the thread waits for the window to scroll and expand; if there is a sequence number in the FREE state, the thread takes the smallest FREE sequence number from the window and updates its state to RESERVED (pre-allocated but not yet sent).

[0059] Since only one thread is responsible for processing the same AccountId at any given time, the retrieval of the serial number for that account is naturally executed serially, eliminating the need for complex distributed lock contention among multiple threads. This significantly improves allocation efficiency while ensuring correctness.

[0060] After obtaining the serial number, the thread merges the serial number with the metadata assembled in the preprocessing stage to construct a complete blockchain transaction. The transaction payload encapsulates at least the following metadata: BizId; AccountId digest; Type; Ts; and request verification information.

[0061] After the transaction is assembled, the thread submits the transaction to the blockchain node. If the blockchain node returns a successful reception, the thread updates the status of the corresponding sequence number to SUBMITTED (sent and awaiting confirmation) and updates the high-water mark in the distributed cache. For example, for AccountId = A001, after being assigned sequence number 109 and successfully broadcast, HighWater[A001] = 109 in the cache.

[0062] After the transaction is submitted, the thread executes the following tracing logic during the subsequent asynchronous confirmation process: If a thread detects an on-chain confirmation event (i.e., the transaction has been packaged into a block and solidified), it updates the status of the sequence number to CONFIRMED. If a sequence number is not confirmed for a long time, and the periodic compensation module finds that the sequence number does not exist in the ledger or the node memory pool during periodic probing, then the sequence number is marked as ABNORMAL and handed over to the compensation module for further analysis and processing. During the continuous allocation of sequence numbers, when the remaining number of allocable windows is lower than a preset threshold (for example, the remaining number of allocable windows is less than 20% of the window size), the baseline sequence number is recalculated based on the latest N1, N2, and N3 values, and the subsequent windows are scrolled forward to expand.

[0063] Using a sliding window instead of requesting the latest sequence number from the blockchain node in real time can significantly reduce the number of RPC round trips and improve the efficiency of transaction sequence number allocation in high-frequency request scenarios; while using third-party reconciliation as the basis for the rolling expansion of the window can prevent the sliding window from being out of touch with the real on-chain state for a long time.

[0064] In an account-based blockchain, if a transaction with a preceding sequence number for an account is missing, subsequent transactions with larger sequence numbers, even if they have entered the node's pending queue, will remain in a pending state for an extended period due to the unmet preceding sequence number, and will not be packaged into a block by the node, forming a typical head-of-queue blockage.

[0065] For example, an account's current ledger confirmation sequence number is 120. The node's pending list contains transactions 122, 123, and 124, but transaction 121 was lost during submission due to network jitter and failed to reach the blockchain node. In this case, transactions with sequences 122, 123, and 124 cannot proceed, and the blockchain node will continuously wait for transaction 121 to arrive before processing subsequent transactions. If relying solely on retrying business messages, the system may not be able to resend the transaction corresponding to sequence 121 in a timely manner. If the original message corresponding to sequence 121 has expired or the processing context has been lost (e.g., the original message in the message queue has been confirmed consumed and deleted), this blocking state will continue to escalate, causing the account's on-chain channel to be completely paralyzed.

[0066] Preferably, to handle the transitional state of account mapping migration during consumer cluster scaling up / down or partition rebalancing, this embodiment also provides a thread migration coordination mechanism.

[0067] When the thread scheduling module determines that a caller account needs to be migrated from the current asynchronous consumer thread (hereinafter referred to as the old thread) to another asynchronous consumer thread (hereinafter referred to as the new thread), the old thread first enters the empty mode. In the empty mode, the old thread stops pulling new messages from the message queue for that account and checks the sequence number status in the local sliding window: If a sequence number with a status of RESERVED exists, the thread will attempt to complete the transaction submission within a preset timeout period; if the submission is not completed within the timeout period, the sequence number will be marked as ABNORMAL. If a sequence number in the SUBMITTED state exists, the thread continues to wait for an acknowledgment event or a timeout determination until all SUBMITTED states change to CONFIRMED or ABNORMAL. When there are no longer any RESERVED or SUBMITTED sequence numbers in the local window, the old thread sends a drain completion signal to the thread scheduling module.

[0068] After receiving the emptying completion signal, the thread scheduling module notifies the new thread to take over the caller's account. The new thread then rebuilds the sequence number sliding window according to the aforementioned three-party reconciliation process.

[0069] The aforementioned emptying mechanism avoids serial number conflicts or stampedes caused by the simultaneous allocation or submission of serial numbers for the same account by both old and new threads during the migration process. For serial numbers marked as ABNORMAL during the emptying process, the timed compensation module performs subsequent filling according to the head-of-line blocking detection procedure described below to ensure the final continuity of the serial number chain.

[0070] In this embodiment, the timed compensation module scans the nodes of key accounts and suspends transaction queues at a preset period (e.g., every 5 seconds, 10 seconds, or 30 seconds; optionally, the preset period can be configured as any value between 1 second and 60 seconds according to the business's sensitivity to latency, with smaller values ​​for latency-sensitive businesses and larger values ​​for throughput-priority businesses). Figure 5 As shown, the specific process is as follows: Retrieve the ledger confirmation sequence number N of account AccountId c ; Retrieve the list of pending transactions for this account {p1, p2, ..., p n Extract the smallest suspended sequence number N. p_min ; If N p_min >N c +1, then determine [N] c +1, N p_min The interval [-1] contains a missing sequence number hole; For each missing sequence number within the interval, construct a corresponding placeholder transaction; The placeholder transaction is sent to the blockchain node to occupy the missing sequence number; Once the hole is filled, previously blocked legitimate transactions (i.e., pending transactions with sequence numbers greater than the hole position) will be activated and can continue to participate in the packaging and confirmation process.

[0071] Optionally, the placeholder transaction can be identified by the built-in logic of the business contract (e.g., by detecting the TX_TYPE=GAP_FILL field), or it can be uniformly received and processed by a dedicated no-operation contract deployed on the chain. Both methods can achieve the unblocking purpose of this invention.

[0072] For example, suppose the ledger confirmation sequence number of an account A001 is currently 52. ​​The asynchronous consumption thread of this account submits three transactions corresponding to sequence numbers 53, 54, and 55 to the blockchain node in sequence. However, due to network routing jitter, the transaction message corresponding to sequence number 53 is lost during transmission and fails to reach the blockchain node; while the transactions corresponding to sequence numbers 54 and 55 successfully reach the node and enter the pending queue in the node's memory pool. At this time, the pending transaction list in the blockchain node's memory pool contains 54 and 55, but lacks 53. Because the blockchain node requires strictly consecutive transaction sequence numbers for the same account, the node places 54 and 55 in the future queue indefinitely, unable to package them into a block. The timed compensation module performs a probe on the account in the next scan cycle (e.g., after 5 seconds), determines that sequence number 53 is missing, and proactively constructs a placeholder transaction with Nonce=53, Payload set to empty (or containing debug flags TX_TYPE=GAP_FILL, MISSING_SEQ=53), and the recipient address set to the no-operation contract address. After the placeholder transaction was sent to the blockchain node, the node found that sequence number 53 was already in use, thus satisfying the continuity condition. Transactions 54 and 55, which were previously suspended, were then moved from the future queue to the pending-packaging queue, and could be packaged and confirmed normally later. At this point, the on-chain channel for this account was fully restored, and subsequent transactions with sequence numbers 56, 57, etc., could be submitted and packaged normally. This example demonstrates that by directly filling missing sequence numbers with placeholder transactions after a gap is detected, the sequence number chain can be quickly restored without restoring the original business content.

[0073] It should be noted that the aforementioned placeholder transaction for smoothing out the flow is not a regular retrieval. A regular retrieval is the repeated transmission of the original business transaction, the purpose of which is to recover lost business data; while the placeholder transaction in this embodiment is not intended to recover the original business content, but to restore the continuity of the account sequence number link by placing the missing sequence number, so that subsequent legitimate transactions can be packaged normally by the node.

[0074] To enhance feasibility and differentiate it from ordinary business transactions, this embodiment imposes the following limitations on the placeholder transaction: The payload is empty or contains only empty operation payload: that is, the Payload field of the placeholder transaction is set to an empty byte array, a fixed placeholder string (such as GAP_FILL), or contains only debug metadata without actual business fields; Includes specific debugging identifier fields: for example, setting TX_TYPE=GAP_FILL (fill type), SOURCE=COMPENSATOR (compensation module source), MISSING_SEQ=121 (missing sequence number), TRACE_ID=xxxx (trace ID) in the transaction payload for subsequent maintenance traceability; The recipient address is a pre-defined no-operation contract address: The no-operation contract refers to a smart contract pre-deployed on the blockchain, used to receive placeholder transactions. This contract verifies the validity of the transaction signature, and upon successful verification, directly returns a success status code without modifying any on-chain business state, writing business data, or triggering any business events. This contract can be pre-deployed at a fixed address on the blockchain, and the recipient address is directly specified as this address when constructing the placeholder transaction; On-chain execution of non-written business states: that is, placeholder transactions are only used to fill in the missing sequence number, are not used as objects of actual business data processing, and are not written into any business-related on-chain states.

[0075] Preferably, the code logic of the no-operation contract is as follows: it only verifies the validity of the transaction signature; upon successful verification, it directly returns a success status code without modifying any on-chain business state, writing any business data, or triggering any business events. This contract can be pre-deployed at a fixed address on the blockchain, and the recipient address is directly specified as this address when constructing the no-operation transaction. Since this contract does not accept any business call parameters and its execution logic is fixed, it ensures that all placeholder transactions sent to this address will not have any side effects on the business state.

[0076] Preferably, the debug flag field is encapsulated in the pre-metadata segment of the transaction payload. At the beginning of the execution logic, the on-chain smart contract first parses the transaction payload and checks whether the pre-metadata segment contains a preset debug flag (e.g., TX_TYPE=GAP_FILL). If the flag is detected, the contract skips all business logic execution and directly returns a success status; if the flag is not detected, it continues to execute normal business logic. This parsing process does not involve changes to business data and serves only as a basis for traffic routing. For example, if implemented using Solidity, the detection can be accomplished by reading the first N bytes of the transaction payload and comparing them with a predefined flag constant.

[0077] To prevent empty transactions from being included in subsequent business statistics, this embodiment performs off-chain marking processing on placeholder transactions already on the blockchain. Specifically, the off-chain index service (i.e., a background service process running independently of the blockchain nodes) obtains the transaction payloads of transactions already on the blockchain by listening to new block events of the blockchain nodes or periodically pulling block data. If a preset debugging identifier field (e.g., TX_TYPE=GAP_FILL) is detected in the transaction payload, a compensation placeholder tag is added to the transaction in the local index database, and a compensation log is recorded. The log fields include at least: account identifier, missing sequence number, placeholder transaction hash, hole detection time, and gap filling time. During business queries, statistical reports, or audit exports, the placeholder transactions are removed from the valid business dataset based on the compensation placeholder tag, while the complete compensation log is retained for operation and maintenance traceability.

[0078] In this embodiment, after a gap is detected, a placeholder transaction is used to fill the missing sequence number, thereby quickly clearing the blockage at the head of the queue. This is particularly crucial for high-frequency transaction pipelines on the blockchain, because such scenarios often focus more on the continuous progress of a large number of subsequent transactions, rather than having to recover the original business load at each interrupted position.

[0079] To prevent the same business request from being executed multiple times on the blockchain due to network retries, duplicate deliveries, or consumer responses, this embodiment maintains the processed state of the global business idempotency identifier BizId in the on-chain smart contract. However, if permanent cumulative storage is adopted (i.e., all processed BizIds are retained in the on-chain state), as the business scale continues to grow, this anti-replay state record will expand indefinitely, leading to an increasing addressing depth of the state tree, continuously rising storage costs, and a gradual decline in read performance, ultimately dragging down the operating efficiency of the blockchain nodes.

[0080] To address this, this embodiment introduces a rolling state mapping tree mechanism based on time epochs, which enables the anti-replay state to have time boundaries and pruning capability, thereby controlling the on-chain storage cost at a fixed addressing depth.

[0081] The time epoch can be defined in the following ways: by block height (e.g., every 1000 blocks is one epoch); by timestamp (e.g., every 1 hour, 6 hours, or 24 hours is one epoch); or by a combination of business batch cycle and block height.

[0082] Preferably, for the timestamp-based implementation, the smart contract can calculate the current epoch number as follows: Let the current block timestamp be T, and the preset epoch duration be D, then the current epoch number EpochId = floor(T / D). This calculation method ensures that the system rolls into the next epoch after a fixed duration.

[0083] Preferably, for the implementation method that combines business batch cycle and block height, the epoch number can be calculated jointly based on the business batch identifier and block height: EpochId = hash(BatchId) XOR floor(H / K), where BatchId is the batch identifier predefined by the business system, H is the current block height, and K is the preset number of blocks. This method can take into account both the semantic boundaries of business batches and the immutable time anchor of the blockchain, and is suitable for business scenarios that require aligning the on-chain anti-replay state with the offline settlement cycle.

[0084] The smart contract internally maintains an EpochState mapping structure organized by epoch number. Each epoch corresponds to an independent state mapping tree, which is used to store the processed BizId digest values ​​within that epoch.

[0085] In this embodiment, the trie is implemented at the underlying level by a Merkle-type trie corresponding to the state database of the blockchain platform. Specifically, the root hash of each epoch tree serves as its unique root pointer and is stored in the top-level EpochState mapping. Because the Merkle-type trie uses a root hash to reference child nodes, all states of any epoch tree can be independently addressed through its root hash, and state sharing between different epochs is only associated through the top-level mapping, thus achieving physical isolation between epochs.

[0086] The pointer / reference relationship between the current epoch tree and the historical epoch tree is jointly maintained through the following two mechanisms: (1) Root hash reference: The smart contract does not store a complete copy of the tree structure, but saves the root hash of each epoch tree as a state variable in the EpochState mapping, with the key being the epoch number and the value being the root hash of that epoch tree.

[0087] (2) Epoch Index Queue: The contract maintains an additional queue of fixed length (using a circular array or linked list structure), which stores the number of the currently surviving epoch and its corresponding root hash in sequence. The queue is arranged in ascending order of epochs (the head of the queue is the oldest epoch, and the tail of the queue is the current epoch). The contract can use this queue to locate any surviving epoch tree in constant time complexity without having to traverse and search through the entire state.

[0088] like Figure 6 As shown, after receiving a business transaction, the smart contract executes the following anti-replay verification process: Extract the global business idempotency identifier BizId from the transaction payload; Perform a hash calculation on BizId to obtain a fixed-length digest key; Look up the digest key in the state map of the current epoch (EpochState[CurrentEpoch]); If the current era is not matched, a limited backtracking search is performed in the preserved historical eras in order from the most recent to the furthest (for example, only the previous M eras are backtracked, M=7), and the search is performed in each historical era tree in turn; If the digest key already exists in any epoch, it is determined that the business request has been processed, and the contract directly returns success (idempotent return), skipping the business execution logic, thereby avoiding repeated execution; If no match is found in any of the queried epochs, the request is determined to be a new request, the business logic is executed, and the digest key of the BizId is written into the state mapping of the current epoch.

[0089] Since each epoch has the same structure, and is limited to the current epoch and a limited number of historical epochs (for example, only the most recent 7 epochs are retained), the overall addressing depth is constrained to a fixed constant (M+1) and does not increase with the growth of the total traffic volume.

[0090] When the cumulative number of epochs reaches a preset threshold N, the system triggers an asynchronous pruning process. In this embodiment, the most recent M epochs are retained (M=7, i.e., the current epoch and the previous 6 historical epochs). When the total number of epochs reaches N=M+1 (i.e., 8 epochs), the state of the oldest epoch is destroyed.

[0091] The cutting process is as follows: Locating the oldest epoch: Retrieve the first element from the epoch index queue and obtain the oldest epoch number OldEpochId and its corresponding root hash; Disconnect the root reference (logical deletion): The contract executes `delete EpochState[OldEpochId]`, removing the root hash of the oldest epoch tree from the top-level `EpochState` mapping. Since any subtree in a Merkle-type trie can only be accessed through its root hash, this operation makes all state data under the oldest epoch no longer accessible at the contract logic level. Clean up the index queue: Pop the queue node corresponding to OldEpochId from the epoch index queue to release the slot it occupies; Physical storage reclamation: After disconnecting root references and cleaning the index queue, the oldest epoch tree has been deleted at the contract level. Storage reclamation is automatically handled by the compression mechanism of the underlying blockchain state database (e.g., a storage engine based on an LSM-Tree architecture): when the storage engine performs compression, leaf nodes and data blocks no longer referenced by any active root node are marked as free space and reclaimed. This process is executed asynchronously by the underlying storage engine, with a time complexity of O(1).

[0092] Preferably, if the blockchain platform supports active root node reference counting or state pruning interfaces, the contract can also call the corresponding system interface when pruning is triggered to accelerate recycling.

[0093] As an alternative, the destruction method can also be implemented using any of the following methods: Mark the state of this era as unreachable (logical deletion); The state of this era will be moved to the history archive area and will no longer be subject to online replay protection verification.

[0094] Compared to existing solutions that trigger rotation based on storage capacity thresholds, the rolling pruning solution based on time epochs emphasizes the consistency between business time windows and blockchain block progression. That is, for business requests exceeding a preset time window (e.g., 7 days), replay protection checks are no longer necessary: ​​application-layer retry and compensation mechanisms are typically only effective within a limited time window (e.g., 24 to 48 hours). Duplicate requests exceeding this window can be considered new business requests or judged and allowed by the business side using business identifiers. If specific business scenarios require auditing and tracing duplicate requests within the timeout window, this can be achieved as follows: upon pruning triggering, the root hash or digest data of the rolling state mapping tree of the expired epoch is exported and archived to an off-chain audit system. The off-chain system retains a complete history of business idempotent identifiers to support subsequent audit queries, but no longer participates in on-chain online replay protection checks. This approach maintains bounded on-chain replay protection storage while also addressing audit tracing requirements.

[0095] Due to the immutability of blockchain, transaction receipts and event logs in the old blocks still exist after the historical epoch tree is deleted. However, the anti-replay verification of this invention is only used to intercept current duplicate transactions. For old business requests that have exceeded a preset time window (e.g., more than the most recent 7 epochs), the application-layer retry mechanism is invalid, and the business side should reconcile and audit through the archiving system. Therefore, destroying expired anti-replay records on the chain does not affect the traceability of historical data.

[0096] The above mechanism can prevent on-chain state explosion caused by the infinite growth of idempotent business states while maintaining a fixed retrieval depth.

[0097] Example 2 The following is a complete business process example to illustrate the collaborative work of the five stages.

[0098] Suppose that in a high-frequency trading platform, account A001 generates 5 transactions to be uploaded to the blockchain in a short period of time, with corresponding BizIds of B101, B102, B103, B104, and B105 respectively.

[0099] Step 1: Request to write to the message queue and schedule to a thread based on account hash. All five requests, each carrying AccountId=A001, were written to the message queue and then mapped to asynchronous thread T7 via consistent hashing during the consumption scheduling phase. This ensures that asynchronous thread T7 becomes the sole sequence number allocator for A001, guaranteeing that requests for the same account are processed serially by a single thread at any given time.

[0100] Step 2: Asynchronous thread T7 is initialized and three-way reconciliation is performed. Assuming the ledger confirms sequence number 50, the memory pool already contains sequences 51 and 52 awaiting packaging, and the cache high-water mark is 51, then the three-party reconciliation yields: N1=50, N2=52, N3=51, N... base =max(50,52,51)=52.

[0101] Therefore, the asynchronous thread T7 reconstructs the window to [53, 152] (with the window size set to W=100).

[0102] Step 3: Pre-assign serial numbers and submit transactions Thread T7 assigns sequence number 53 to B101, 54 to B102, and 55 to B103, submits them to the blockchain node in sequence, and synchronously updates the cache high watermark to 55.

[0103] Step 4: Missing serial number causes queue head blocking. Suppose that transaction corresponding to sequence number 53 of B101 fails to enter the memory pool due to a momentary failure in node reception (e.g., network timeout preventing the node from receiving the transaction), while sequences 54 and 55 corresponding to B102 and B103 have successfully entered the node's pending queue. At this point, the ledger confirmation is still stuck at sequence number 52, and the smallest sequence number in the pending list is 54, indicating that sequence number 53 is missing, and sequences 54 and 55 are blocked.

[0104] Step 5: The timed compensation module detects voids and initiates concurrent empty transactions. The compensation module detected N p_min =54>N c +1=53, thus identifying the missing sequence number 53. The system constructs a compensation placeholder transaction: Nonce: 53; Payload: null or {"TX_TYPE":"GAP_FILL","MISSING_SEQ":53}; Receiver: null operation contract address.

[0105] After the placeholder transaction entered the node, it successfully occupied sequence number 53. Subsequently, the legitimate transactions corresponding to the previously blocked sequence numbers 54 and 55 gained the prerequisites to continue being packaged, the node began to process these subsequent transactions normally, and the system channel was automatically restored.

[0106] Step Six: On-chain Replay Protection When the transaction corresponding to B102 is actually executed, the contract retrieves BizId=B102, first searches for it in the current epoch state mapping tree. If it does not exist, the normal business logic is executed, and the digest of B102 is written to the current epoch state mapping. If B102 is submitted again due to duplicate message queue delivery (even if its sequence number may have been reassigned), the contract will still determine it as a duplicate request because BizId already exists in the epoch mapping tree, directly return success, and skip business execution, thus ensuring idempotency at the business level.

[0107] Step 7: Historical State Cropping As the system runs for a long time, BizIds accumulate continuously. Contracts always retain only the anti-replay mappings for the most recent 7 epochs, performing asynchronous destruction operations on earlier epochs. Therefore, even if the total business volume reaches hundreds of millions, the storage size and addressing depth of the anti-replay records remain constant and will not expand with business growth.

[0108] As can be seen from the above process, the present invention solves the concurrency source problem through the following five levels of collaborative control: account-level thread scheduling solves the rebalancing and recovery problem; third-party reconciliation solves the high-frequency pre-allocation efficiency problem; sliding window solves the suspension and blocking problem; placeholder transaction unblocking solves the problem of repeated execution and state explosion; thus forming a complete technical closed loop that adapts to the asynchronous on-chain scenario of message queues.

[0109] Preferably, during the implementation of this invention, if the number of requests from a specific caller account exceeds a preset threshold, adaptive adjustments can be made using one or more of the following methods: (1) At the business level, the account is split into multiple logical sub-accounts, and each sub-account independently executes the method steps of the present invention to improve the concurrent processing capability while maintaining the order of transaction sequence numbers; (2) Configure a larger sliding window capacity and a higher system resource priority for the fixed asynchronous consumption thread routed to this account; (3) At the asynchronous consumption thread pool configuration level, increase the message prefetching quantity for the dedicated asynchronous consumption thread of high-load account to reduce the time the thread is idle while waiting for messages.

[0110] like Figure 7 As shown, the blockchain data upload system based on message queues provided by this invention includes the following modules: The request access module is used to receive data uplink requests carrying a global business idempotency identifier and a caller account identifier, and to persist them to a message queue. The hash grouping scheduling module is used to perform consistent hash routing based on the caller account identifier, and schedule data uploading requests from the same caller account to a fixed asynchronous consumption thread; The three-party reconciliation and window reconstruction module is used to synchronously obtain the maximum confirmed transaction sequence number in the blockchain ledger, the maximum sequence number to be packaged in the node memory pool, and the high-water mark in the distributed cache when the asynchronous consumption thread is initialized and / or when partition rebalancing, thread migration, or fault recovery occurs in the message queue, and reconstruct the sequence number sliding window of the account with the maximum value of the three. The sequence number pre-allocation and on-chain submission module is used to pre-allocate strictly increasing transaction sequence numbers from the sequence number sliding window, construct a blockchain transaction containing a global business idempotent identifier and submit it to the blockchain node, while updating the high-water mark in the distributed cache. The thread scheduling module is used to control the migration of the caller account from the current asynchronous consumer thread to the target asynchronous consumer thread during the scaling up or down of the consumer cluster or partition rebalancing. During the migration process, it controls the old thread to perform the emptying operation and controls the new thread to take over and trigger the three-party reconciliation and reconstruction after the emptying is completed. The timed compensation module is used to periodically detect the pending transaction queues of each account, identify the queue head blockage caused by missing preceding transaction sequence numbers, and construct placeholder transactions to clear the blockage. The on-chain anti-replay execution module is used to perform existence checks on the global business idempotent flag in the rolling state mapping tree based on time epoch levels. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the repeated business execution is blocked, and the historical mapping of the expired epoch is periodically pruned and destroyed. The data flow and control flow relationships between the above modules are as follows: Figure 7 As indicated by the arrows, the specific functions of each module have been described in detail in Embodiment 1, and will not be repeated here.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for uploading blockchain data based on a message queue, characterized in that, include: Receive data upload requests carrying a global business idempotency identifier and a caller account identifier and persist them to the message queue; Based on the caller account identifier, consistent hash routing is performed to schedule data uplink requests from the same caller account to a fixed asynchronous consumption thread for serial processing, thereby constraining the allocation order of the account's transaction sequence number; When the asynchronous consumption thread is initialized and / or when partition rebalancing, thread migration or fault recovery occurs in the message queue, it synchronously obtains the confirmed maximum transaction sequence number in the blockchain ledger, the maximum transaction sequence number to be packaged in the node memory pool and the high watermark in the distributed cache, and reconstructs the sequence number sliding window of the account with the maximum value of the three. The asynchronous consumption thread pre-allocates strictly increasing transaction sequence numbers from the sequence number sliding window, generates a blockchain transaction containing the global business idempotent identifier, submits it to the blockchain node, and updates the high-water mark in the distributed cache at the same time. The timed compensation module periodically probes the pending transaction queues of each account, identifies queue head blockages caused by missing preceding transaction sequence numbers, and proactively constructs placeholder transactions with missing transaction sequence numbers to send to the blockchain node in order to clear the blocked subsequent transactions. The on-chain smart contract extracts the global business idempotency identifier and performs an existence check in the rolling state mapping tree based on time epochs. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the execution of repeated business is blocked. Furthermore, when the rolling state mapping tree reaches the preset number of epochs to be retained, the historical mappings of the expired epochs are periodically pruned and destroyed.

2. The method according to claim 1, characterized in that, Performing consistent hashing routing based on the caller's account identifier includes: Using the caller account identifier as hash input and combining it with the consumer thread set identifier, the target slot is calculated, mapping the same caller account identifier to the same asynchronous consumer thread; When the number of message queue partitions, the number of consumer instances, or the set of threads changes, only the caller account of the affected slot is migrated; The step of scheduling data upload requests from the same caller account to a fixed asynchronous consumption thread is to avoid sequence number confusion, overwriting, or conflict caused by multiple threads allocating transaction sequence numbers in parallel when the same account is consuming concurrently.

3. The method according to claim 1, characterized in that, The sliding window that reconstructs the account's serial number using the maximum of the three values ​​includes: Query the maximum transaction sequence number of the most recently confirmed on-chain transaction of the calling account mentioned in the blockchain ledger, and denote it as N1; Query the maximum transaction sequence number of the transactions of the caller account that are pending packaging, suspended or propagated in the blockchain node memory pool, and denote it as N2; Read the high-water mark of the transaction that the caller account has been allocated or committed, recorded in the distributed cache, and denote it as N3; Take max(N1, N2, N3) as the reconstruction baseline value N. base and in accordance with [N] base+1 N base+W Initialize the serial number sliding window in local memory, where W is the window size.

4. The method according to claim 3, characterized in that, The sliding window reconstruction is triggered in any of the following scenarios: After the asynchronous consumer thread starts, it takes over the caller's account for the first time. A message queue partition rebalancing caused the caller account to migrate from the current asynchronous consumer thread to another asynchronous consumer thread; The asynchronous consumption thread detected anomalies such as loss of sequence number allocation status in local memory, rollback of cache status, and blockchain node returning sequence numbers that are too low or too high. Asynchronous consumer thread can recover from crashes, network interruptions, or process hot migrations.

5. The method according to claim 3, characterized in that, Pre-allocating strictly incremental transaction sequence numbers from the sequence number sliding window includes: For the same caller account, only the target asynchronous consumer thread is allowed to claim the smallest unallocated sequence number in the current sliding window in sequence; After the transaction is constructed and sent to the blockchain node, the status corresponding to the assigned sequence number will be updated from "assignable" to "submitted and awaiting confirmation". Upon receiving a blockchain node receipt, on-chain confirmation event, or compensation probe result, the status of the corresponding sequence number will be updated to confirmed or abnormal and awaiting compensation. When the assigned sequence number approaches the upper boundary of the sliding window, the next window interval is rolled out based on the latest third-party reconciliation results.

6. The method according to claim 1, characterized in that, The global business idempotency identifier is encapsulated in a business metadata field within the transaction payload, and the business metadata field includes at least: One or more of the following fields: global business idempotency identifier, request creation time, caller account identifier summary, business type identifier, and request signature verification field; The on-chain smart contract parses the business metadata fields and uses the global business idempotent identifier as a unique key to perform anti-replay verification.

7. The method according to claim 1, characterized in that, The timed compensation module periodically detects the pending transaction queues of each account, including: Retrieve the list of pending transactions of the account in the blockchain node according to a preset scanning cycle; Identify head-of-line blocking holes based on gaps in consecutive transaction sequence numbers in the pending transaction list; When the minimum pending transaction sequence number is detected to be greater than the ledger confirmation sequence number plus 1, it is determined that there is a queue head blockage caused by the loss of the preceding transaction sequence number; A placeholder transaction is constructed for the missing sequence number of the blockage hole at the head of the queue and sent to the blockchain node.

8. The method according to claim 7, characterized in that, The placeholder transaction is an empty transaction that does not trigger actual business execution, and the empty transaction satisfies at least one of the following: The transaction payload is empty. The transaction payload includes a preset debugging identifier field, which is used to indicate that the transaction is a placeholder transaction; The transaction recipient address is either a preset no-operation contract address or a system reserved address; The transaction execution logic only completes the sequence number placeholder and returns a success status, without writing any business data.

9. The method according to claim 1, characterized in that, The periodic pruning and destruction of historical mappings of outdated epochs includes: When the cumulative number of epochs reaches the preset threshold N, an asynchronous pruning task is triggered; Retain the state mappings corresponding to the most recent M epochs, where N is greater than M and M is a positive integer; Perform unbinding, deletion, or unreachability processing on the root node of the historical dictionary tree that is earlier than the most recent M epochs to release on-chain state occupancy and block repeated access paths of overdue idempotent identifiers.

10. A blockchain data on-chain system based on message queues, characterized in that, include: The request access module is used to receive data uplink requests carrying a global business idempotency identifier and a caller account identifier and persist them to a message queue. The hash grouping scheduling module is used to perform consistent hash routing based on the caller account identifier, and schedule data uploading requests from the same caller account to a fixed asynchronous consumption thread; The third-party reconciliation and window reconstruction module is used to synchronously obtain the confirmed maximum transaction sequence number in the blockchain ledger, the maximum sequence number to be packaged in the node memory pool, and the high-water mark in the distributed cache when the asynchronous consumption thread is initialized and / or when partition rebalancing, thread migration, or fault recovery occurs in the message queue, and reconstruct the sequence number sliding window of the account with the maximum value. The sequence number pre-allocation and on-chain submission module is used to pre-allocate strictly increasing transaction sequence numbers from the sequence number sliding window, construct blockchain transactions containing global business idempotency identifiers, and update the high-water mark in the distributed cache. The thread scheduling module is used to control the migration of the caller account from the current asynchronous consumer thread to the target asynchronous consumer thread during the scaling up or down of the consumer cluster or partition rebalancing. During the migration process, it controls the old thread to perform the emptying operation and controls the new thread to take over and trigger the three-party reconciliation and reconstruction after the emptying is completed. The timed compensation module is used to periodically detect the pending transaction queues of each account, identify the queue head blockage caused by missing preceding transaction sequence numbers, and construct placeholder transactions to clear the blockage. The on-chain anti-replay execution module is used to perform existence checks on the global business idempotent identifier in the rolling state mapping tree based on time epoch hierarchy. If it does not exist, the business logic is executed and written into the current epoch state mapping. If it already exists, the repeated business execution is blocked, and the historical mapping of the expired epoch is periodically pruned and destroyed.

Images