A data security union processing method and system
By employing dynamic sharding, distributed Bloom filters, and multi-threaded parallel merging techniques, the problems of data skew and hash collisions in the merging process of ultra-large datasets are solved, achieving efficient and low-latency dataset merging.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUXI ANNIWELL TECHNOLOGY CO LTD
- Filing Date
- 2025-04-11
- Publication Date
- 2026-07-10
Smart Images
Figure CN120371864B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data management technology, and in particular to a data security union processing method and system. Background Technology
[0002] Mega-datasets are collections of data on a massive scale. Due to their sheer size and rapid, continuously growing volume, coupled with limitations in system memory and disk resources, they are difficult to store in traditional relational databases. Therefore, when performing association analysis on two related mega-datasets, the common practice is to merge them first. This transforms the complex association analysis that would otherwise require two related datasets into retrieval, classification, and statistical operations on a single dataset, effectively reducing the difficulty of extracting information from mega-datasets.
[0003] In practice, when merging two related, very large datasets, smaller datasets are typically stored in a database first, while the larger dataset is split into multiple data shards and merged in parallel with the datasets in the database. However, this process has the following main problems: First, in existing technologies, the sharding results may exhibit data skew, meaning a large amount of data is concentrated on one or more small shards for merging calculations, while the remaining small amount of data is distributed across a large number of shards. Since the overall efficiency of the merging calculation depends on the computational efficiency of all shards, and the merging efficiency of a small number of shards is far below the average, this significantly reduces the overall merging efficiency. Second, during the merging process, it is necessary to traverse the data in each data shard one by one. Each time a data record is read, an access request needs to be sent to the database to find matching data. This results in an excessive number of database requests, increasing network pressure. Furthermore, since most databases have a limited number of requests, this frequent requesting greatly limits the improvement of the overall dataset merging efficiency.
[0004] To address the aforementioned issues, the method and apparatus for merging ultra-large datasets disclosed in CN109033295B, with the key technical points being: converting the first associated primary key of the first dataset into data of a preset field type, then fragmenting the first dataset according to the first associated primary key, obtaining a preset number of first data fragments, and caching them in a preset cache system; converting the second associated primary key of the second dataset into data of a preset field type, then fragmenting the second dataset according to the second associated primary key, obtaining a preset number of second data fragments; reading the first data fragments from the preset cache system, matching the first data fragments and the second data fragments, and merging the matched first data fragments and second data fragments.
[0005] However, in the above methods, since hash collisions are an inherent defect of hash algorithms and cannot be completely avoided, introducing cross-shard retrieval will destroy the parallelism of shards, significantly increase the merging complexity, and cause the actual related data to be scattered in different shards due to hash collisions when matching shard numbers, thus requiring cross-shard retrieval.
[0006] Therefore, a data security union processing method and system are proposed to solve or alleviate the above problems. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a secure data union processing method and system.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A data security union processing method includes the following steps:
[0010] Dynamic sharding divides the dataset into independent sharding pools for high-frequency primary keys and multi-level hash sharding for low-frequency primary keys based on the data frequency of the primary keys.
[0011] Conflict pre-detection uses a distributed Bloom filter to predict potential conflicting primary keys and generates a list of alternative shards;
[0012] Local redirection involves sending targeted query requests for conflicting primary keys only to shards in their spare shard list during the merge phase.
[0013] Parallel merging: Initiate multi-threaded parallel merging of the primary shard data and incrementally aggregate conflicting query results;
[0014] Load balancing, real-time monitoring of shard load, dynamic migration of overloaded shards and adjustment of hash parameters.
[0015] Preferably, the dynamic sharding, based on the data frequency of the primary keys, divides the dataset into a high-frequency primary key independent sharding pool and a low-frequency primary key multi-level hash sharding pool, specifically including the following steps:
[0016] Statistical analysis of primary key frequency distribution identifies primary keys whose frequency exceeds a preset threshold as high-frequency primary keys;
[0017] Each high-frequency primary key is assigned an independent sharding pool, and a consistent hashing algorithm is used to map all data of the same high-frequency primary key to the same shard.
[0018] Low-frequency primary keys are sharded by applying a multi-level hash function chain in sequence. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a primary key collides after sharding with H1, sharding with H2 is automatically triggered. If H2 still results in a collision, sharding with H3 is triggered.
[0019] The mapping relationship between high-frequency primary keys and shard pools, and the multi-level hash sharding rules for low-frequency primary keys are written into a distributed routing table based on LSM-Tree. The routing table is stored in partitions according to shard numbers, and data consistency is ensured through WAL.
[0020] Preferably, the conflict pre-detection, which uses a distributed Bloom filter to predict potential conflicting primary keys and generate a list of alternative shards, specifically includes the following steps:
[0021] Each shard maintains a Bloom filter, using 3 independent hash functions;
[0022] Hash the primary key, map the result to a bit array of length m, set the false positive rate of the Bloom filter to p, and then apply the formula... Determine the length of the bit array, where n is the number of primary keys within the slice;
[0023] The Bloom filter bit array of each shard is periodically synchronized to other nodes via the Gossip protocol, with a synchronization period of T.
[0024] The CRDT merging algorithm is used to resolve bit array conflicts during the synchronization process;
[0025] During the sharding phase, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, a list of alternative shards is generated based on the results of the multi-level hash chain.
[0026] The list of spare shards is sorted by priority, with the priority rule being: H3 shard > H2 shard > H1 shard.
[0027] Preferably, the local redirection, during the merging phase, involves sending targeted query requests to shards in the spare shard list only for conflicting primary keys, specifically including the following steps:
[0028] Write the potentially conflicting primary key and its alternative shard list into a lightweight conflict table. The key is a byte-type long value of the primary key, and the value is a list of alternative shard numbers in JSON format. The lightweight conflict table is implemented based on RedisCluster, and the conflict table TTL is set to twice the merge task timeout.
[0029] If the primary shard fails to match during the merge phase, the conflict table is queried to obtain a list of alternative shards.
[0030] Send batch query requests to the backup shard, with each request containing up to 100 primary keys;
[0031] If the backup shard query still returns no results, the primary key data is discarded and recorded in the exception log.
[0032] Preferably, the parallel merging involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflict query results, specifically including the following steps:
[0033] The merging task of a single shard is split into multiple subtasks. Each subtask processes a continuous primary key range. Each subtask starts an independent thread and the task allocation is achieved through a lock-free queue.
[0034] The primary shard merge result is written to a Redis Sorted Set according to the shard number. The Score is a long value of the primary key, and the Value is the merged data record.
[0035] Conflict query results are batch-appended to the corresponding Sorted Set via Redis Pipeline;
[0036] After the merge is complete, export the Sorted Set data in primary key order to generate the final dataset.
[0037] Preferably, the parallel merging, which involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflict query results, further includes the following steps:
[0038] At the moment the merge is completed, a snapshot is generated for each data fragment based on the Chandy-Lamport algorithm;
[0039] Compare snapshot data of the primary shard and the standby shard to identify mismatched primary keys;
[0040] For missing primary keys, a secondary merging process is triggered, with a maximum retry count of 3. If the secondary merging still fails, the primary key is marked as "unmergeable data" and pushed to the manual review queue.
[0041] Preferably, the load balancing, which involves real-time monitoring of shard load, dynamic migration of overloaded shards, and adjustment of hash parameters, specifically includes the following steps:
[0042] Collect data sharding metrics such as data volume, query latency, and conflict rate.
[0043] If the amount of data in a shard exceeds a preset threshold or the conflict rate is greater than 5%, a shard split or migration is triggered, splitting it into two sub-shards. The sub-shard rules follow the original multi-level hash chain. During the migration, the old and new shards serve query requests in parallel, and data consistency is ensured through a dual-write mechanism. After the migration is completed, the distributed routing table is updated and the old shard data is cleaned up.
[0044] Preferably, the load balancing, which involves real-time monitoring of shard load, dynamic migration of overloaded shards, and adjustment of hash parameters, further includes the following steps:
[0045] The priority of hash functions is dynamically adjusted based on historical collision rates. If the collision rate of H1 exceeds 10% for three consecutive times, H2 will be automatically set as the primary hash function.
[0046] The hash seed is periodically optimized by using a simulated annealing algorithm to find the optimal seed value.
[0047] Preferably, the load balancing, which involves real-time monitoring of shard load, dynamic migration of overloaded shards, and adjustment of hash parameters, further includes the following steps:
[0048] The number of shards is dynamically adjusted based on the total data volume D and the maximum capacity of a single shard C, according to the formula... Dynamically calculate the number of fragments;
[0049] If the current number of shards is less than N, trigger the shard expansion process and redistribute data according to the multi-level hash chain rules;
[0050] The collision rate feedback control establishes a PID controller with the real-time collision rate as the input and the hash seed adjustment amount as the output. If the collision rate exceeds the set threshold, the hash function parameters are updated using the gradient descent algorithm.
[0051] The present invention also provides a data security union processing system for performing the operation instructions of the above method when executed, including a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module via a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.
[0052] The present invention has the following beneficial effects:
[0053] This invention utilizes a high-frequency primary key independent sharding pool and a multi-level hash chain to distribute low-frequency data, a distributed Bloom filter for collision pre-detection, and local redirection query technology. Combined with multi-threaded parallel processing and real-time load balancing optimization, it limits the proportion of cross-shard retrieval caused by hash collisions. While ensuring shard parallelism, it reduces the hash collision rate, enabling efficient, low-latency, and highly complete merging of ultra-large datasets. This avoids the problem that shard number matching may result in actual related data being scattered across different shards due to hash collisions, requiring cross-shard retrieval. Attached Figure Description
[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0057] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0058] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0059] In the description of this invention, it should be understood that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used to facilitate the description of this invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0060] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0061] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0062] A data security union processing method includes the following steps:
[0063] Dynamic sharding divides the dataset into independent shard pools for high-frequency primary keys and multi-level hash shards for low-frequency primary keys based on the data frequency of the primary keys. Most conflicts are resolved within the shards through multi-level hash chains, eliminating the need for cross-shard operations.
[0064] Conflict pre-detection uses a distributed Bloom filter to predict potentially conflicting primary keys and generate a list of alternative shards. It quickly identifies primary keys that may conflict and triggers subsequent processing only for primary keys with pre-detected conflicts, avoiding full data traversal.
[0065] Local redirection sends targeted query requests to shards in the standby shard list only for conflicting primary keys during the merging phase, limiting the cross-shard range and restricting cross-shard retrieval to the standby shard list rather than the full shard list. Targeted queries are executed in parallel with primary shard merging, avoiding serial waiting.
[0066] Parallel merging starts multi-threaded parallel merging of the main shard data and incrementally aggregates conflict query results for fast conflict resolution. During merging, the conflict table is directly queried to obtain the list of alternative shards, reducing real-time computing overhead. Furthermore, the metadata is lightweight, avoiding storage explosion caused by recording all data. In addition, parallelization within shards makes up for the extra time consumption of targeted queries, and the total merging time is close to that of pure parallel shard matching.
[0067] Load balancing, real-time monitoring of shard load, dynamic migration of overloaded shards and adjustment of hash parameters.
[0068] Preferably, dynamic sharding divides the dataset into independent sharding pools for high-frequency primary keys and multi-level hash sharding for low-frequency primary keys based on the data frequency of the primary keys. Specifically, this includes the following steps:
[0069] Statistical analysis of primary key frequency distribution identifies primary keys whose frequency exceeds a preset threshold as high-frequency primary keys;
[0070] Each high-frequency primary key is assigned an independent sharding pool, and a consistent hashing algorithm is used to map all data of the same high-frequency primary key to the same shard.
[0071] Low-frequency primary keys are sharded by applying a multi-level hash function chain in sequence. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a primary key sharding collision occurs after H1 sharding, H2 sharding is automatically triggered. If H2 still results in a collision, H3 sharding is triggered.
[0072] The mapping relationship between high-frequency primary keys and shard pools, and the multi-level hash sharding rules for low-frequency primary keys are written into a distributed routing table based on LSM-Tree. The routing table is stored in partitions according to shard numbers, and data consistency is guaranteed through WAL.
[0073] Preferably, conflict pre-detection involves using a distributed Bloom filter to predict potentially conflicting primary keys and generating a list of alternative shards, specifically including the following steps:
[0074] Each shard maintains a Bloom filter, using 3 independent hash functions;
[0075] Hash the primary key, map the result to a bit array of length m, set the false positive rate of the Bloom filter to p, and then apply the formula... Determine the length of the bit array, where n is the number of primary keys within the slice;
[0076] The Bloom filter bit array of each shard is periodically synchronized to other nodes via the Gossip protocol, with a synchronization period of T.
[0077] The CRDT merging algorithm is used to resolve bit array conflicts during the synchronization process;
[0078] During the sharding phase, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, a list of alternative shards is generated based on the results of the multi-level hash chain.
[0079] The list of spare shards is sorted by priority, with the priority rule being: H3 shard > H2 shard > H1 shard.
[0080] Preferably, partial redirection involves sending targeted query requests for conflicting primary keys only to shards in their spare shard list during the merge phase. This specifically includes the following steps:
[0081] Write the potentially conflicting primary keys and their alternative shard lists into a lightweight conflict table. The key is a byte-type long value of the primary key, and the value is a list of alternative shard numbers in JSON format. The lightweight conflict table is implemented based on Redis Cluster, and the conflict table TTL is set to twice the merge task timeout.
[0082] If the primary shard fails to match during the merge phase, the conflict table is queried to obtain a list of alternative shards.
[0083] Send batch query requests to the backup shard, with each request containing up to 100 primary keys;
[0084] If the backup shard query still returns no results, the primary key data is discarded and recorded in the exception log.
[0085] Preferably, parallel merging involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflicting query results, specifically including the following steps:
[0086] The merging task of a single shard is split into multiple subtasks. Each subtask processes a continuous primary key range. Each subtask starts an independent thread and the task allocation is achieved through a lock-free queue.
[0087] The primary shard merge result is written to a Redis Sorted Set according to the shard number. The Score is a long value of the primary key, and the Value is the merged data record.
[0088] Conflict query results are batch-appended to the corresponding Sorted Set via Redis Pipeline;
[0089] After the merge is complete, export the Sorted Set data in primary key order to generate the final dataset.
[0090] Preferably, the parallel merging process involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflicting query results, and also includes the following steps.
[0091] At the moment the merge is completed, a snapshot is generated for each data fragment based on the Chandy-Lamport algorithm;
[0092] Compare snapshot data of the primary shard and the standby shard to identify mismatched primary keys;
[0093] For missing primary keys, a secondary merging process is triggered, with a maximum retry count of 3. If the secondary merging still fails, the primary key is marked as "unmergeable data" and pushed to the manual review queue.
[0094] Preferably, load balancing involves real-time monitoring of shard load, dynamic migration of overloaded shards, and adjustment of hash parameters, specifically including the following steps:
[0095] Collect data sharding metrics such as data volume, query latency, and conflict rate.
[0096] If the amount of data in a shard exceeds a preset threshold or the conflict rate is greater than 5%, a shard split or migration is triggered, splitting it into two sub-shards. The sub-shard rules follow the original multi-level hash chain. During the migration, the old and new shards serve query requests in parallel, and data consistency is ensured through a dual-write mechanism. After the migration is completed, the distributed routing table is updated and the old shard data is cleaned up.
[0097] Preferably, load balancing, real-time monitoring of shard load, dynamic migration of overloaded shards and adjustment of hash parameters, further includes the following steps:
[0098] The priority of hash functions is dynamically adjusted based on historical collision rates. If the collision rate of H1 exceeds 10% for three consecutive times, H2 will be automatically set as the primary hash function.
[0099] The hash seed is periodically optimized by using a simulated annealing algorithm to find the optimal seed value.
[0100] Preferably, load balancing, real-time monitoring of shard load, dynamic migration of overloaded shards and adjustment of hash parameters, further includes the following steps:
[0101] The number of shards is dynamically adjusted based on the total data volume D and the maximum capacity of a single shard C, according to the formula... Dynamically calculate the number of fragments;
[0102] If the current number of shards is less than N, trigger the shard expansion process and redistribute data according to the multi-level hash chain rules;
[0103] The collision rate feedback control establishes a PID controller with the real-time collision rate as the input and the hash seed adjustment amount as the output. If the collision rate exceeds the set threshold, the hash function parameters are updated using the gradient descent algorithm.
[0104] This invention achieves efficient and complete data integration through dynamic fragmentation and conflict awareness technology. Its specific working process is as follows:
[0105] First, the dataset is dynamically sharded. Based on primary key frequency analysis, high-frequency primary keys are independently allocated to a dedicated sharding pool to avoid conflicts. Low-frequency primary keys are distributed to different shards through a multi-level hash chain, applying the MurmurHash algorithm, CityHash algorithm and XXHash algorithm in sequence. The sharding rules and hash parameters are recorded in real time in a distributed routing table based on LSM-Tree.
[0106] Subsequently, a distributed Bloom filter is used to perform conflict pre-detection. Each shard maintains a local Bloom filter and synchronizes a global conflict view with the Gossip protocol. After identifying potential conflicting primary keys, a priority-sorted list of alternative shards is generated, with shard H3 taking precedence over shards H2 and H1. The mapping relationship is then written to a lightweight conflict table built by Redis Cluster.
[0107] During the merge phase, the primary shard data is processed in parallel by multiple threads and split into subtasks based on the primary key range, which are then divided into CPU cores × 2. The tasks are allocated using a lock-free queue, and the results are written to the RedisSortedSet cache in real time. If the primary shard matching fails, a list of backup shards is extracted from the conflict table, and a batch targeted query is initiated, with a maximum of 100 primary keys per query. The conflict results are incrementally appended to the cache via the Pipeline, and the consistency between the primary and backup shards is verified through the Chandy-Lamport snapshot mechanism. Any missing data triggers a secondary merge process.
[0108] Throughout the process, the system monitors the shard load in real time. When the data volume exceeds 1TB or the conflict rate exceeds 5%, it automatically splits the overloaded shards and migrates the data. At the same time, it dynamically optimizes the hash seed and the number of shards based on the simulated annealing algorithm, and adjusts the conflict rate through a closed loop using a PID controller. Ultimately, while ensuring parallelism, the cross-shard retrieval ratio is limited to less than 1%, and the conflict rate is reduced to less than 0.1%, achieving efficient and stable ultra-large-scale data merging.
[0109] The present invention also provides a data security union processing system for performing the operation instructions of the above method when executed, including a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module via a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.
[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A data security union processing method, characterized in that, Includes the following steps: Dynamic sharding divides the dataset into independent sharding pools for high-frequency primary keys and multi-level hash sharding for low-frequency primary keys based on the data frequency of the primary keys. Statistical analysis of primary key frequency distribution identifies primary keys whose frequency exceeds a preset threshold as high-frequency primary keys; Each high-frequency primary key is assigned an independent sharding pool, and a consistent hashing algorithm is used to map all data of the same high-frequency primary key to the same shard. Low-frequency primary keys are sharded by applying a multi-level hash function chain in sequence. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a primary key collides after sharding with H1, sharding with H2 is automatically triggered. If H2 still results in a collision, sharding with H3 is triggered. The mapping relationship between high-frequency primary keys and sharding pools, and the multi-level hash sharding rules of low-frequency primary keys are written into a distributed routing table based on LSM-Tree. The routing table is stored in partitions according to shard numbers, and data consistency is guaranteed by WAL. Conflict pre-detection uses a distributed Bloom filter to predict potential conflicting primary keys and generates a list of alternative shards; Local redirection involves sending targeted query requests for conflicting primary keys only to shards in their spare shard list during the merge phase. Parallel merging: Initiate multi-threaded parallel merging of the primary shard data and incrementally aggregate conflicting query results; Load balancing, real-time monitoring of shard load, dynamic migration of overloaded shards and adjustment of hash parameters.
2. The data security union processing method according to claim 1, characterized in that, The conflict pre-detection, which uses a distributed Bloom filter to predict potential conflicting primary keys and generate a list of alternative shards, specifically includes the following steps: Each shard maintains a Bloom filter, using 3 independent hash functions; Hash the primary key, map the result to a bit array of length m, set the false positive rate of the Bloom filter to p, and then apply the formula... Determine the length of the bit array, where n is the number of primary keys within the slice; The Bloom filter bit array of each shard is periodically synchronized to other nodes via the Gossip protocol, with a synchronization period of T. The CRDT merging algorithm is used to resolve bit array conflicts during the synchronization process; During the sharding phase, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, a list of alternative shards is generated based on the results of the multi-level hash chain. The list of spare shards is sorted by priority, with the priority rule being: H3 shard > H2 shard > H1 shard.
3. The data security union processing method according to claim 1, characterized in that, The local redirection, during the merging phase, involves sending targeted query requests to shards in the spare shard list only for conflicting primary keys. Specifically, it includes the following steps: Write the potentially conflicting primary keys and their alternative shard lists into a lightweight conflict table. The key is a byte-type long value of the primary key, and the value is a list of alternative shard numbers in JSON format. The lightweight conflict table is implemented based on Redis Cluster, and the conflict table TTL is set to twice the merge task timeout. If the primary shard fails to match during the merge phase, the conflict table is queried to obtain a list of alternative shards. Send batch query requests to the backup shard, with each request containing up to 100 primary keys; If the backup shard query still yields no results, the primary key data is discarded and recorded in the exception log.
4. The data security union processing method according to claim 1, characterized in that, The parallel merging process involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflict query results. Specifically, it includes the following steps: The merging task of a single shard is split into multiple subtasks. Each subtask processes a continuous primary key range. Each subtask starts an independent thread and the task allocation is achieved through a lock-free queue. The primary shard merge result is written to a Redis Sorted Set according to the shard number. The Score is a long value of the primary key, and the Value is the merged data record. Conflict query results are batch-appended to the corresponding Sorted Set via Redis Pipeline; After the merge is complete, export the Sorted Set data in primary key order to generate the final dataset.
5. A data security union processing method according to claim 4, characterized in that, The parallel merging process involves initiating multi-threaded parallel merging of the primary shard data and incrementally aggregating conflict query results. It also includes the following steps: At the moment the merge is completed, a snapshot is generated for each data fragment based on the Chandy-Lamport algorithm; Compare snapshot data of the primary shard and the standby shard to identify mismatched primary keys; For missing primary keys, a secondary merging process is triggered, with a maximum retry count of 3. If the secondary merging still fails, the primary key is marked as "unmergeable data" and pushed to the manual review queue.
6. The data security union processing method according to claim 1, characterized in that, The load balancing mechanism monitors shard load in real time, dynamically migrates overloaded shards, and adjusts hash parameters. Specifically, it includes the following steps: Collect data sharding metrics such as data volume, query latency, and conflict rate. If the amount of data in a shard exceeds a preset threshold or the conflict rate is greater than 5%, a shard split or migration is triggered, splitting it into two sub-shards. The sub-shard rules follow the original multi-level hash chain. During the migration, the old and new shards serve query requests in parallel, and data consistency is ensured through a dual-write mechanism. After the migration is completed, the distributed routing table is updated and the old shard data is cleaned up.
7. A data security union processing method according to claim 6, characterized in that, The load balancing mechanism, which monitors shard load in real time, dynamically migrates overloaded shards and adjusts hash parameters, also includes the following steps: The priority of the hash function is dynamically adjusted based on the historical collision rate. If the collision rate of H1 exceeds 10% for three consecutive times, H2 will be automatically set as the main hash function. The hash seed is periodically optimized by using a simulated annealing algorithm to find the optimal seed value.
8. A data security union processing method according to claim 7, characterized in that, The load balancing mechanism, which monitors shard load in real time, dynamically migrates overloaded shards and adjusts hash parameters, also includes the following steps: The number of shards is dynamically adjusted based on the total data volume D and the maximum capacity of a single shard C, according to the formula... Dynamically calculate the number of fragments; If the current number of shards is less than N, trigger the shard expansion process and redistribute data according to the multi-level hash chain rules; The collision rate feedback control uses a PID controller with the real-time collision rate as input and the hash seed adjustment as output. If the collision rate exceeds a set threshold, the hash function parameters are updated using a gradient descent algorithm.
9. A data security union processing system, wherein, when executed, it performs operation instructions for the data security union processing method as described in claims 1 to 8, characterized in that, It includes a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module via a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.