A blockchain transaction numbering-oriented deduplication filtering method and system
By using a distributed Bloom filter and persistence mechanism, the memory bloat and performance bottleneck issues of deduplication of blockchain transaction numbers are solved, achieving efficient and reliable deduplication filtering, which is suitable for high-throughput blockchain systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHOUSHAN DONGHAI MICROCORE OCEAN DIGITAL SCIENCE INSTITUTE
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies for deduplicating blockchain transaction numbers suffer from memory consumption bloat and performance bottlenecks, especially with ultra-large-scale transaction data, making it impossible to achieve fast, accurate, and persistent deduplication filtering.
A distributed Bloom filter architecture is adopted, which combines pre-write logging and filter state persistence mechanism. Transaction numbers are distributed to multiple filter nodes through hash sharding, and a batch processing mechanism is introduced to achieve efficient deduplication filtering.
While ensuring high concurrency and high throughput, it achieves fast querying and persistent storage of transaction numbers, reduces memory usage and network interaction overhead, and supports the stable deduplication requirement of billions of transactions.
Smart Images
Figure CN122152809A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital data processing technology, and more specifically to a deduplication filtering method and system for blockchain transaction numbers. Background Technology
[0002] Currently, various technical methods have been adopted in the industry to solve the problem of deduplicating transaction numbers (txid). The most direct method is to maintain a hash set (such as using an in-memory Set or similar structure) to store all seen txids, and check their existence in the set each time a new transaction arrives. The average lookup complexity of a hash set is O(1), which is fast. However, its disadvantage is that it requires storing all txids completely, resulting in a space complexity of O(N). As the number of transactions increases, the memory occupied by the hash set will expand rapidly, and billions of transactions may exhaust the system memory. In addition, a pure in-memory set has poor reliability if it is not persisted separately, and the data will be lost once the system crashes.
[0003] Another approach is to store each transaction number as a key in a persistent database (such as LevelDB or a relational database), and check for existence using the database index when querying for a new txid. This method offers persistence because the data is stored on disk, avoiding memory limitations. However, its performance bottleneck lies in the fact that query operations slow down when the data volume becomes massive.
[0004] Therefore, how to achieve fast, accurate, and persistent deduplication filtering of transaction numbers under ultra-large-scale transaction data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of the above problems, the present invention is proposed to provide a method and system for deduplication filtering of blockchain transaction numbers that overcomes or at least partially solves the above problems.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, embodiments of the present invention provide a deduplication filtering method for blockchain transaction numbers, characterized in that it includes: S1. Set up one management node and multiple filter nodes, and deploy a Bloom filter on each filter node; S2. The management node receives the transaction number to be deduplicated and routes the transaction number to the corresponding filtering node according to the preset hash sharding rules. S3. The filtering node performs an existence query and records the transaction IDs routed to it: If the query result does not exist, the transaction number is determined to be a new transaction, the transaction number is inserted into the Bloom filter, and the corresponding persistent log record is generated; If the query result shows that the transaction number exists, it is determined to be a duplicate transaction, and the transaction number is prevented from being inserted into the Bloom filter.
[0008] Furthermore, the specific process of generating persistent log records in S3 is as follows: Persistent logging includes a pre-write log file and an append-only operation log file; When a transaction number is determined to be a new transaction, the transaction number is recorded in the pre-write log file, and the status update operation of the Bloom filter is synchronously recorded in the append operation log file corresponding to the filter node.
[0009] Furthermore, when the system restarts due to a failure, all processed transaction number records are restored by replaying the pre-write log file, and the memory state of the Bloom filter of each filter node is reconstructed by loading the append-only operation log file.
[0010] Furthermore, the pre-write log file uses a sequential disk write method.
[0011] Furthermore, it also includes batch processing support: When the management node receives a batch of transaction numbers, it groups the batch of transaction numbers according to the hash sharding rule to obtain a set of transaction numbers. Distribute the set of transaction IDs to the corresponding filter nodes in parallel; The filter node receives the set of transaction numbers and performs batch query and batch insert operations.
[0012] Furthermore, the parameters of the Bloom filter can be adjusted. The system sets the array length and number of hash functions for each Bloom filter based on the estimated maximum number of transactions and the allowable false positive rate.
[0013] Furthermore, when the number of transactions continues to grow or it is necessary to reduce the false positive rate, the capacity is expanded by increasing the number of filtering nodes, and the transaction number hash sharding mapping relationship is reassigned by the management node.
[0014] Furthermore, the management node employs a consistent hashing algorithm to implement hash sharding rules.
[0015] Furthermore, a deduplication filtering system for blockchain transaction numbers includes a management node module and multiple filtering node modules; The management node module is used to receive the transaction number to be deduplicated and to route the transaction number to the corresponding filter node module according to the preset hash sharding rules. The management node module communicates with multiple filter node modules; Each filter node module includes: Bloom filter example: used to perform existence queries and record entries for transaction numbers in memory; Log Management Unit: When a query result does not exist, the transaction number is determined to be a new transaction, the transaction number is inserted into the Bloom filter, and the corresponding persistent log record is generated. If the query results exist, the transaction number is determined to be a duplicate transaction, and that transaction number is prevented from being inserted into the Bloom filter.
[0016] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: 1. This invention provides a method and system for deduplication filtering of blockchain transaction numbers. By using a Bloom filter to perform member detection on the transaction number, efficient deduplication filtering is achieved without storing the original transaction number.
[0017] 2. This invention adopts a distributed filtering architecture based on transaction number hash sharding, which distributes a large set of transaction numbers to multiple filtering nodes according to hash rules, thereby breaking through the limitations of single-node memory and computing power.
[0018] 3. This invention introduces a mechanism that combines pre-write logging with filter state persistence, which ensures the performance of memory-level deduplication queries while achieving reliable and persistent storage of deduplication state and crash recovery.
[0019] 4. This invention supports batch query and batch write processing of transaction numbers, which can significantly reduce network interaction and scheduling overhead and improve the overall throughput performance in block-level or high-concurrency scenarios.
[0020] 5. The deduplication filtering performance of this invention is basically decoupled from the transaction scale. Even when the number of transaction numbers continues to grow, the query and insertion time complexity can still be kept approximately constant.
[0021] 6. This invention is applicable to transaction processing scenarios of high-throughput, long-term blockchain systems, and can stably support the deduplication filtering requirements of more than one billion transaction numbers under limited resource conditions. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0023] Figure 1 This is a flowchart of the overall technical solution provided in the embodiments of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] This invention discloses a deduplication filtering method and system for blockchain transaction numbers, such as... Figure 1 As shown, it includes: S1. Set up one management node and multiple filter nodes, and deploy a Bloom filter on each filter node; S2. The management node receives the transaction number to be deduplicated and routes the transaction number to the corresponding filtering node according to the preset hash sharding rules. S3. The filtering node performs an existence query and records the transaction IDs routed to it: If the query result does not exist, the transaction number is determined to be a new transaction, the transaction number is inserted into the Bloom filter, and the corresponding persistent log record is generated; If the query result shows that the transaction number exists, it is determined to be a duplicate transaction, and the transaction number is prevented from being inserted into the Bloom filter.
[0026] The specific implementation of this invention is as follows: Deduplication filtering process: When the management node receives a new transaction txid, it forwards it to the corresponding filter node for processing according to the hash routing algorithm described above. The Bloom filter on the filter node first performs a member lookup: it hashes the txid k times, calculates the corresponding k addresses, and checks whether these positions in the bit array are all 1. If any bit is 0, it means that the txid has never appeared before and is judged as a new transaction; if all bits are 1, it means that the txid may already exist (there is a possibility of false positives). For txids judged as new transactions, the filter node will set the corresponding bits to 1 using the same k hashes, completing the operation of inserting the txid into the Bloom filter. Thus, when the same txid is received again, the filter will give a "may exist" result, thereby achieving the purpose of deduplication. The query and update operations of the Bloom filter are completed in memory in constant time, and the time for a single deduplication check is only in the microsecond range, which can support real-time filtering under high concurrency. Once a transaction is determined to be a duplicate (all corresponding bits are 1), it means that the system has previously processed the txid. At this point, the method of this invention will prevent the transaction from entering the subsequent process (such as refusing it to enter the pool or package), effectively preventing double payment or duplicate transaction records from occurring.
[0027] Persistent Log Mechanism: To address the need for persistent storage of deduplicated data, this invention introduces a two-layer persistence mechanism to ensure data reliability. First, whenever a txid identified as a new transaction needs to be inserted into the filter, the system appends this txid to the Write-Ahead Log (WAL) file. The WAL uses sequential disk writes, which have low overhead and ensure data is written to disk before completion, so even if the system subsequently crashes, the record of the new txid will not be lost. Simultaneously, since the filter nodes operate on an in-memory database, they also utilize their own persistence capabilities to periodically or in real-time synchronize the Bloom filter state to disk (e.g., Redis's AOF append file mechanism). The WAL records a streaming log of txids, while the AOF records the update operation log of the Bloom filter bit arrays; the combination of the two constitutes a robust persistence solution. When the system fails and restarts, all txid records up to the last moment before the failure can be recovered by replaying the WAL logs. Then, each filter node can load the most recent filter bit array state from its respective AOF file to quickly reconstruct the entire deduplication filter's in-memory image. This dual protection ensures zero data loss and rapid recovery: even if a single node crashes or the entire system restarts, it can automatically recover to the deduplication state before the crash upon startup and continue to provide seamless service.
[0028] Batch Processing Support: While supporting real-time filtering of single transactions, the method of this invention also provides batch processing capabilities to match the application scenarios of blockchain-packaged transactions. The system allows for one-time deduplication checks or batch insertions of a batch of txids. For example, when a block is generated, all the transaction txids contained within can be submitted for filtering together. The management node will send this batch of txids to the respective target filtering nodes according to hash shards, and each node will perform query or insertion operations in parallel. Compared to processing transaction by transaction, batch processing amortizes network communication and scheduling overhead, significantly improving throughput. Based on an existing 5 billion txids, batch checking 10,000 new txids takes only about 30 milliseconds, averaging only 3 microseconds per transaction; batch inserting 10,000 txids takes about 10 milliseconds, averaging 1 microsecond per transaction. This performance demonstrates that this solution can easily handle the input rate of tens of thousands of transactions per second in real-world blockchain networks, achieving high-throughput deduplication filtering.
[0029] Adjustable Parameters and Scalability Strategies: This invention provides various parameters and strategies to adapt to networks of different sizes and control the false positive rate. First, the Bloom filter parameters (bit array size and number of hash functions) are adjustable: the system can set the bit array length and number of hash functions for each filter based on the estimated maximum number of transactions and the allowable false positive rate, achieving a balance between space usage and false positive rate. Larger bit arrays or more hash functions reduce the false positive rate but increase memory / computational overhead; smaller filters have the opposite effect. Operators can flexibly adjust these parameters according to actual needs, thereby controlling the false positive rate within an acceptable range (e.g., reducing the false positive rate to well below 1%). Second, the system has good horizontal scalability: when the number of transactions continues to grow or when further reduction in the false positive rate is needed, the system capacity can be expanded by increasing the number of filter nodes. Adding nodes (i.e., adding shards) is equivalent to splitting the global txid set more finely, reducing the amount of data processed by each Bloom filter instance, thus also reducing the saturation and false positive rate of a single instance. The management node can seamlessly integrate new filtering nodes into the cluster using strategies such as consistent hashing and rebalance txid mappings to avoid single points of failure. This scalable design ensures that the system can be scaled up as needed, theoretically supporting arbitrarily large transaction sets. Finally, the management node itself can be redundantly or distributed in the architecture to avoid the risk of single points of failure caused by centralized coordination, thereby further improving the robustness and availability of the system.
[0030] Based on the same inventive concept, this embodiment of the invention also provides a deduplication filtering system for blockchain transaction numbers. Since the principle of solving the problem by these devices and clients is similar to that of the aforementioned deduplication filtering method for blockchain transaction numbers, the implementation of these devices and clients can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.
[0031] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0032] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A deduplication filtering method for blockchain transaction numbers, characterized in that, include: S1. Set up one management node and multiple filter nodes, and deploy a Bloom filter on each filter node; S2. The management node receives the transaction number to be deduplicated and routes the transaction number to the corresponding filtering node according to the preset hash sharding rules. S3. The filtering node performs an existence query and records the transaction number routed to the node: If the query result is not found, the transaction number is determined to be a new transaction, the transaction number is inserted into the Bloom filter, and a corresponding persistent log record is generated; If the query result shows that the transaction number exists, it is determined that the transaction number is a duplicate transaction, and the transaction number is prevented from being inserted into the Bloom filter.
2. The method as described in claim 1, characterized in that, The specific process of generating persistent log records in S3 is as follows: The persistent log records include a pre-write log file and an append-only operation log file; When a transaction number is determined to be a new transaction, the transaction number is recorded in the pre-write log file, and the status update operation of the Bloom filter is synchronously recorded in the append operation log file corresponding to the filter node.
3. The method as described in claim 2, characterized in that, Also includes: When the system restarts due to a failure, all processed transaction number records are restored by replaying the pre-write log file, and the memory state of the Bloom filter of each filter node is reconstructed by loading the append-only operation log file.
4. The method as described in claim 2, characterized in that, The pre-write log file is written to disk sequentially.
5. The method as described in claim 1, characterized in that, It also includes batch processing support: When the management node receives a batch of transaction numbers, it groups the batch of transaction numbers according to the hash sharding rule to obtain a set of transaction numbers. The set of transaction numbers is distributed in parallel to the corresponding filter nodes; The filter node receives the set of transaction numbers and performs batch query and batch insert operations.
6. The method as described in claim 1, characterized in that, include: The parameters of the Bloom filter can be adjusted. The system sets the array length and number of hash functions for each Bloom filter based on the estimated maximum number of transactions and the allowable false positive rate.
7. The method as described in claim 1, characterized in that, Also includes: When the number of transactions continues to grow or it is necessary to reduce the false positive rate, the capacity is expanded by increasing the number of filtering nodes, and the management node reallocates the transaction number hash sharding mapping relationship.
8. The method as described in claim 1, characterized in that, include: The management node uses a consistent hashing algorithm to implement the hash sharding rules.
9. A deduplication filtering system for blockchain transaction numbers, characterized in that, The method for implementing any one of claims 1 to 8 includes a management node module and multiple filter node modules; The management node module is used to receive the transaction number to be deduplicated and to route the transaction number to the corresponding filtering node module according to the preset hash sharding rules. The management node module is communicatively connected to multiple filter node modules; Each of the filter node modules includes: Bloom filter example: used to perform existence queries and record entries for transaction numbers in memory; Log management unit: When the query result does not exist, it determines that the transaction number is a new transaction, inserts the transaction number into the Bloom filter, and generates the corresponding persistent log record; If the query result exists, the transaction number is determined to be a duplicate transaction, and the transaction number is prevented from being inserted into the Bloom filter.