Data security union set processing method and system

Abstract

The invention discloses a data security union set processing method and system, belongs to the technical field of data management, and solves the problems that hash conflicts are inherent defects of a hash algorithm and cannot be completely avoided, fragment parallelism can be damaged by introduction of cross-fragment retrieval, merging complexity is remarkably increased, and data security is poor. And therefore, actual associated data may be dispersed in different fragments due to hash conflicts in fragment number matching, and cross-fragment retrieval is needed. Comprising the following steps: dynamically fragmenting according to the data popularity of a primary key; through dynamic fragmentation, distributed bloom filter conflict pre-detection and local redirection query technologies, multi-thread parallel processing and real-time load balancing optimization are combined, the cross-fragmentation retrieval proportion caused by hash conflicts is limited, the hash conflict rate is reduced while the fragmentation parallelism is guaranteed, super-large data sets are merged, and the retrieval efficiency is improved. And the problem that in fragment number matching, actual associated data are possibly dispersed in different fragments due to Hash conflicts, and cross-fragment retrieval is needed is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular, to a data security union processing method and system. Background Art

[0002] A super-large dataset is a collection composed of datasets with a large amount of data volume. Due to its extremely large data volume itself, and the data volume is still growing rapidly, coupled with the limitations of resources such as system memory and disk, it is difficult to store the super-large dataset in a traditional relational database. Therefore, when performing correlation analysis on two related super-large datasets, the common approach is to first merge these two datasets. In this way, the complex correlation analysis that originally needed to be performed on two related datasets can be transformed into retrieval, classification, and statistical operations on a single dataset, thereby effectively reducing the difficulty of mining information from the super-large dataset.

[0003] In actual operation, when merging two related super-large datasets, usually some smaller datasets are first stored in the database, and the larger datasets are split into multiple data shards and merged in parallel with the datasets in the database. However, there are the following main problems in this process: First, in the prior art, the sharding result of the dataset may have a data skew phenomenon, that is, a large amount of data is concentrated on one or more small shards for merging calculation, while the remaining small amount of data is distributed on a large number of shards for merging calculation. Since the overall efficiency of the merging calculation depends on the calculation efficiency of all shards, and the merging calculation efficiency of a small number of shards is much lower than the average value, this significantly reduces the overall merging efficiency. Second, during the merging process, it is necessary to traverse each data shard data item by item. For each piece of data read, a database access request needs to be sent to find the matching data. This results in too many requests to the database, increasing the network pressure. At the same time, due to the limited number of requests of most databases, this frequent request greatly limits the improvement of the overall merging efficiency of the dataset.

[0004] To solve the above problems, the merging method and device for super-large datasets with the publication number CN109033295B have the technical key points that the first associated primary key of the first dataset is converted into data of a preset field type, and then the first dataset is sharded according to the first associated primary key to obtain a preset number of first data shards and cache them in a preset cache system; the second associated primary key of the second dataset is converted into data of a preset field type, and then the second dataset is sharded according to the second associated primary key to obtain a preset number of second data shards; the first data shards are read from the preset cache system, the first data shards and the second data shards are matched, and the matched first data shards and second data shards are merged.

[0005] However, in the above method, since hash conflict is an inherent defect of the hash algorithm and cannot be completely avoided, introducing cross-shard retrieval will destroy the shard parallelism, significantly increase the merging complexity, and cause the shard number matching to be scattered in different shards due to hash conflicts, requiring cross-shard retrieval.

[0006] Therefore, a data security union processing method and system are proposed to solve or alleviate the above problems. Summary of the Invention

[0007] The purpose of the present invention is to solve the defects existing in the prior art, and a data security union processing method and system are proposed.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] A data security union processing method includes the following steps:

[0010] Dynamic sharding, dividing the data set into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard according to the data heat of the primary key;

[0011] Conflict pre-check, predicting potential conflict primary keys through a distributed Bloom filter and generating a list of backup shards;

[0012] Local redirection, sending a directed query request only to the shards in the backup shard list of the conflict primary key during the merging stage;

[0013] Parallel merging, starting multi-threaded parallel merging of the main shard data and incrementally aggregating the conflict query results;

[0014] Load balancing, monitoring the shard load in real time, dynamically migrating overloaded shards and adjusting the hash parameters.

[0015] Preferably, the dynamic sharding, dividing the data set into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard according to the data heat of the primary key, specifically includes the following steps:

[0016] Statistical primary key frequency distribution, identifying the primary keys with frequencies exceeding the preset threshold as high-frequency primary keys;

[0017] Allocating an independent shard pool for each high-frequency primary key, and using the consistent hash algorithm to map all data of the same high-frequency primary key to the same shard;

[0018] Applying a multi-level hash function chain to the low-frequency primary keys for sharding in sequence. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a conflict occurs after sharding by H1, H2 sharding is automatically triggered. If H2 still conflicts, H3 sharding is triggered;

[0019] Write the mapping relationship between the high-frequency primary keys and the shard pool, and the multi-level hash sharding rules for the low-frequency primary keys into the distributed routing table based on the LSM-Tree. The routing table is stored partitioned by shard number, and data consistency is ensured through WAL.

[0020] Preferably, for the conflict pre-check, a distributed Bloom filter is used to predict potential conflicting primary keys and generate a list of alternative shards. Specifically, it includes the following steps:

[0021] Each shard maintains a Bloom filter using 3 independent hash functions;

[0022] Hash the primary key, map the result to a bit array of length m, set the false positive rate of the Bloom filter to p, and determine the length of the bit array according to the formula where n is the number of primary keys within the shard;

[0023] Periodically synchronize the Bloom filter bit arrays of each shard to other nodes through the Gossip protocol, with a synchronization period of T;

[0024] Use the CRDT merge algorithm to resolve the conflicts of the bit arrays during the synchronization process;

[0025] During the sharding stage, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, generate a list of alternative shards according to the multi-level hash chain result;

[0026] The list of alternative shards is sorted by priority, and the priority rule is: Shard H3 > Shard H2 > Shard H1.

[0027] Preferably, for the local redirection, during the merge stage, only send a directed query request for the conflicting primary key to the shards in its list of alternative shards. Specifically, it includes the following steps:

[0028] Write the potential conflicting primary keys and their list of alternative shards into a lightweight conflict table. The key is the byte-type long value of the primary key, and the value is a list of alternative shard numbers in JSON format. The lightweight conflict table is implemented based on RedisCluster, and the conflict table TTL is set to 2 times the merge task timeout time;

[0029] During the merge stage, if the main shard matching fails, query the conflict table to obtain the list of alternative shards;

[0030] Send a batch query request to the alternative shards, with a maximum of 100 primary keys in a single request;

[0031] If no result is obtained from the query of the alternative shards, discard the primary key data and record it in the exception log.

[0032] Preferably, for the parallel merging, multi-threaded parallel merging is initiated for the main shard data, and the conflict query results are incrementally aggregated. The specific steps are as follows:

[0033] Split the merging task of a single shard into multiple sub-tasks. Each sub-task processes a continuous primary key range, and each sub-task starts an independent thread. The task assignment is achieved through a lock-free queue.

[0034] The main shard merging results are written into Redis Sorted Set according to the shard number. The Score is the long value of the primary key, and the Value is the merged data record.

[0035] The conflict query results are batch appended to the corresponding Sorted Set through Redis Pipeline.

[0036] After the merging is completed, export the Sorted Set data in the order of the primary key to generate the final data set.

[0037] Preferably, for the parallel merging, multi-threaded parallel merging is initiated for the main shard data, and the conflict query results are incrementally aggregated. It further includes the following steps

[0038] At the instant when the merging is completed, generate snapshots for the data of each shard based on the Chandy-Lamport algorithm.

[0039] Compare the snapshot data of the main shard and the standby shard to identify the unmatched primary keys.

[0040] Trigger a secondary merging process for the missing primary keys, with the maximum retry limit set to 3 times. If the secondary merging still fails, mark the primary key as "data that cannot be merged" and push it to the manual review queue.

[0041] Preferably, for the load balancing, the shard load is monitored in real time, overloaded shards are dynamically migrated, and the hash parameters are adjusted. The specific steps are as follows:

[0042] Collect the data volume, query latency, and conflict rate metrics of the shards.

[0043] If the data volume of a shard exceeds the preset threshold or the conflict rate > 5%, trigger shard splitting or migration. Split it into two sub-shards, and the sub-shard rules follow the original multi-level hash chain. During the migration process, the old and new shards serve query requests in parallel, and the data consistency is ensured through the double-write mechanism. After the migration is completed, update the distributed routing table and clean up the old shard data.

[0044] Preferably, for the load balancing, the shard load is monitored in real time, overloaded shards are dynamically migrated, and the hash parameters are adjusted. It further includes the following steps:

[0045] Dynamically adjust the priority of the hash function according to the historical conflict rate. If the conflict rate of H1 exceeds 10% continuously for 3 times, then automatically set H2 as the main hash function;

[0046] Periodically optimize the hash seed, and use the simulated annealing algorithm to find the optimal seed value.

[0047] Preferably, for the load balancing, it monitors the shard load in real time, dynamically migrates overloaded shards and adjusts the hash parameters, and further includes the following steps:

[0048] Dynamically adjust the number of shards. According to the total data volume D and the upper limit C of the single shard capacity, dynamically calculate the number of shards according to the formula; Dynamically calculate the number of shards;

[0049] If the current number of shards < N, trigger the shard expansion process, and re - allocate the data according to the multi - level hash chain rule;

[0050] Conflict rate feedback control. Establish a PID controller, with the input being the real - time conflict rate and the output being the adjustment amount of the hash seed. If the conflict rate exceeds the set threshold, update the hash function parameters through the gradient descent algorithm.

[0051] The present invention also provides a data security union processing system for operating instructions of the above - mentioned method when being executed, including a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module through a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.

[0052] The present invention has the following beneficial effects:

[0053] Through the high - frequency primary key independent shard pool and the multi - level hash chain to disperse low - frequency data, the distributed Bloom filter conflict pre - detection and local redirection query technology, combined with multi - thread parallel processing and real - time load balancing optimization, the present invention limits the cross - shard retrieval ratio caused by hash conflicts, reduces the hash conflict rate while ensuring the shard parallelism, performs a super - large data set union with high efficiency, low latency, and high integrity, and avoids the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash conflicts and requires cross - shard retrieval. Description of the Drawings

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, so they should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0055] Figure 1 It is a flow block diagram of the present invention. Detailed implementation mode

[0056] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention usually described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0058] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0059] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed during use, or the orientation or positional relationship commonly understood by those skilled in the art. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0060] In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and cannot be understood as indicating or implying relative importance.

[0061] In the description of the present invention, it should also be noted that unless otherwise clearly specified and defined, the terms "set", "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0062] A data security union processing method includes the following steps:

[0063] Dynamic sharding: According to the data heat of the primary key, the data set is divided into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard. Most conflicts are resolved through a multi-level hash chain within the shard, eliminating the need for cross-shard operations;

[0064] Conflict pre-check: Use a distributed Bloom filter to predict potential conflicting primary keys and generate a list of alternative shards, quickly identifying potentially conflicting primary keys. Only trigger subsequent processing for pre-checked conflicting primary keys to avoid traversing all data;

[0065] Local redirection: During the merge phase, send directed query requests for conflicting primary keys only to the shards in their alternative shard lists, restricting the cross-shard scope. Limit cross-shard retrieval within the alternative shard list instead of all shards. The directed query and the main shard merge are executed in parallel to avoid serial waiting;

[0066] Parallel merge: Start multi-threaded parallel merging of the main shard data and incrementally aggregate the conflict query results for fast conflict resolution. When merging, directly query the conflict table to obtain the alternative shard list, reducing real-time computing overhead. In addition, the metadata is lightweight to avoid storage explosion caused by recording all data. Furthermore, through in-shard parallelization, make up for the additional time consumption of directed queries, and the total merge time is close to pure parallel shard matching;

[0067] Load balancing: Monitor the shard load in real time, dynamically migrate overloaded shards, and adjust the hash parameters.

[0068] Preferably, for dynamic sharding, according to the data heat of the primary key, the data set is divided into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard, which specifically includes the following steps:

[0069] Statistical frequency distribution of primary keys: Identify primary keys with a frequency exceeding a preset threshold as high-frequency primary keys;

[0070] Allocate an independent shard pool for each high-frequency primary key, and use the consistent hashing algorithm to map all data of the same high-frequency primary key to the same shard;

[0071] Apply a multi-level hash function chain to shard low-frequency primary keys in sequence. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a conflict occurs after sharding by H1, H2 sharding is automatically triggered. If H2 still conflicts, H3 sharding is triggered;

[0072] Write the mapping relationship between high-frequency primary keys and shard pools and the multi-level hash sharding rules for low-frequency primary keys into a distributed routing table based on LSM-Tree. The routing table is stored by shard number partition, and WAL is used to ensure data consistency.

[0073] Preferably, for conflict pre-check, a distributed Bloom filter is used to predict potential conflicting primary keys and generate a list of alternative shards. The specific steps are as follows:

[0074] Each shard maintains a Bloom filter and uses 3 independent hash functions;

[0075] Hash the primary key, map the result to a bit array of length m, set the misjudgment rate of the Bloom filter to p, and according to the formula Determine the length of the bit array, where n is the number of primary keys within the shard;

[0076] Periodically synchronize the bit arrays of the Bloom filters of each shard to other nodes through the Gossip protocol, and the synchronization period is T;

[0077] Adopt the CRDT merge algorithm to solve the conflicts of the bit arrays during the synchronization process;

[0078] During the sharding stage, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, generate a list of alternative shards according to the results of the multi-level hash chain;

[0079] The list of alternative shards is sorted by priority, and the priority rule is: Shard H3 > Shard H2 > Shard H1.

[0080] Preferably, for local redirection, during the merge stage, only send a directed query request for the conflicting primary key to the shards in its list of alternative shards. The specific steps are as follows:

[0081] Write the potential conflicting primary key and its list of alternative shards into a lightweight conflict table. The key is the byte-type long value of the primary key, and the value is a list of alternative shard numbers in JSON format. The lightweight conflict table is implemented based on Redis Cluster, and the conflict table TTL is set to 2 times the merge task timeout time;

[0082] If the main shard fails to match during the merge stage, query the conflict table to obtain the list of alternative shards;

[0083] Send a batch query request to the alternative shards, and each single request contains at most 100 primary keys;

[0084] If no result is obtained from the query of the alternative shards, discard the primary key data and record it in the exception log.

[0085] Preferably, for parallel merge, start multi-threaded parallel merge for the main shard data and incrementally aggregate the conflict query results. The specific steps are as follows:

[0086] Split the merge task of a single shard into multiple sub-tasks. Each sub-task processes a continuous primary key range, and each sub-task starts an independent thread to achieve task allocation through a lock-free queue;

[0087] The merged results of the main shards are written into the Redis Sorted Set according to the shard numbers. The Score is the long value of the primary key, and the Value is the merged data record;

[0088] The conflict query results are appended to the corresponding Sorted Set in batches through the Redis Pipeline;

[0089] After the merging is completed, the data in the Sorted Set is exported in the order of the primary keys to generate the final data set.

[0090] Preferably, for parallel merging, multiple threads are started to parallelly merge the main shard data, and the conflict query results are incrementally aggregated. The following steps are also included

[0091] At the moment when the merging is completed, snapshots are generated for the data of each shard based on the Chandy-Lamport algorithm;

[0092] Compare the snapshot data of the main shard and the standby shard to identify the unmatched primary keys;

[0093] Trigger a secondary merging process for the missing primary keys, limit the maximum number of retries to 3 times. If the secondary merging still fails, mark the primary key as "data that cannot be merged" and push it to the manual review queue.

[0094] Preferably, for load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters. Specifically, the following steps are included:

[0095] Collect the data volume, query latency, and conflict rate metrics of the shards;

[0096] If the data volume of a shard exceeds the preset threshold or the conflict rate > 5%, trigger shard splitting or migration. Split it into two sub-shards, and the sub-shard rules follow the original multi-level hash chain. During the migration process, the old and new shards serve query requests in parallel, and ensure data consistency through the dual-write mechanism. After the migration is completed, update the distributed routing table and clean up the data of the old shard.

[0097] Preferably, for load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters. The following steps are also included:

[0098] Dynamically adjust the priority of the hash function according to the historical conflict rate. If the conflict rate of H1 exceeds 10% continuously for 3 times, then automatically set H2 as the main hash function;

[0099] Periodically optimize the hash seed, and use the simulated annealing algorithm to find the optimal seed value.

[0100] Preferably, for load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters. The following steps are also included:

[0101] The number of shards is dynamically adjusted. Based on the total data volume D and the upper limit C of the single shard capacity, the number of shards is dynamically calculated according to the formula ;

[0102] If the current number of shards < N, trigger the shard expansion process, and re - distribute the data according to the multi - level hash chain rule;

[0103] Conflict rate feedback control: Establish a PID controller. The input is the real - time conflict rate, and the output is the adjustment amount of the hash seed. If the conflict rate exceeds the set threshold, update the hash function parameters through the gradient descent algorithm.

[0104] The present invention realizes efficient and complete data integration through dynamic sharding and conflict - aware technology. The specific working process is as follows:

[0105] First, perform dynamic sharding on the data set. Based on the primary key frequency analysis, the high - frequency primary keys are independently assigned to the dedicated shard pool to avoid conflicts, and the low - frequency primary keys are successively dispersed to different shards through the multi - level hash chain, applying the MurmurHash algorithm, CityHash algorithm, and XXHash algorithm in sequence. The sharding rules and hash parameters are recorded in real - time in the distributed routing table based on the LSM - Tree.

[0106] Subsequently, perform conflict pre - detection through the distributed Bloom filter. Each shard maintains a local Bloom filter and synchronizes the global conflict view with the help of the Gossip protocol. After identifying potential conflict primary keys, generate a priority - sorted list of alternative shards. The H3 shard has a higher priority than the H2 and H1 shards, and write the mapping relationship into the lightweight conflict table constructed by the Redis Cluster.

[0107] In the merging stage, the data of the main shard is split into sub - tasks of CPU core number × 2 according to the primary key range through multi - thread parallel processing, and the tasks are allocated using a lock - free queue and the results are written into the Redis SortedSet cache in real - time. If the main shard matching fails, extract the list of alternative shards from the conflict table, initiate a batch - oriented query with a maximum of 100 primary keys at a time, and incrementally append the conflict results to the cache through the Pipeline. Then, verify the consistency of the main and alternative shards through the Chandy - Lamport snapshot mechanism. Missing data triggers the secondary merging process.

[0108] Throughout the process, the system monitors the shard load in real - time. When the data volume exceeds 1TB or the conflict rate exceeds 5%, automatically split the overloaded shards and migrate the data. At the same time, dynamically optimize the hash seed and the number of shards based on the simulated annealing algorithm, and close - loop regulate the conflict rate through the PID controller. Finally, while ensuring parallelism, limit the cross - shard retrieval ratio within 1% and reduce the conflict rate below 0.1%, achieving efficient and stable ultra - large - scale data merging.

[0109] The present invention also provides a data security union processing system, which, when executed, performs the operation instructions of the above method, and includes a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module through a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.

[0110] The foregoing are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data security union processing method, characterized in that, It includes the following steps: Dynamic sharding: According to the data heat of the primary key, the data set is divided into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard; Conflict pre-check: Use a distributed Bloom filter to predict potential conflicting primary keys and generate a list of alternative shards; Local redirection: In the merging stage, only send a directed query request to the shards in the alternative shard list for the conflicting primary key; Parallel merging: Start multi-threaded parallel merging of the primary shard data and incrementally aggregate the conflicting query results; Load balancing: Monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters.

2. The data security union processing method according to claim 1, characterized in that For the dynamic sharding, according to the data heat of the primary key, the data set is divided into a high-frequency primary key independent shard pool and a low-frequency primary key multi-level hash shard, which specifically includes the following steps: Statistical frequency distribution of primary keys, and identify the primary keys with frequencies exceeding the preset threshold as high-frequency primary keys; Allocate an independent shard pool for each high-frequency primary key, and use the consistent hashing algorithm to map all data of the same high-frequency primary key to the same shard; Apply a multi-level hash function chain to the low-frequency primary keys in turn for sharding. The multi-level hash function chain is H1→H2→H3, where H1 is the MurmurHash algorithm, H2 is the CityHash algorithm, and H3 is the XXHash algorithm. If a conflict occurs after sharding by H1, H2 sharding is automatically triggered. If H2 still conflicts, H3 sharding is triggered; Write the mapping relationship between the high-frequency primary key and the shard pool, and the multi-level hash sharding rules of the low-frequency primary key into the distributed routing table based on the LSM-Tree. The routing table is stored by partition according to the shard number, and the data consistency is ensured through WAL.

3. The data security union processing method according to claim 1, characterized in that For the conflict pre-check, use a distributed Bloom filter to predict potential conflicting primary keys and generate a list of alternative shards, which specifically includes the following steps: Each shard maintains a Bloom filter and uses 3 independent hash functions; Hash the primary key, map the result to a bit array of length m, set the false positive rate of the Bloom filter to p, and according to the formula Determine the length of the bit array, where n is the number of primary keys within the shard; Periodically synchronize the Bloom filter bit arrays of each shard to other nodes through the Gossip protocol, and the synchronization period is T; Use the CRDT merging algorithm to solve the bit array conflict during the synchronization process; In the sharding stage, if the hash value of the primary key is marked as existing in multiple shards in the global Bloom filter, generate a list of alternative shards according to the multi-level hash chain result; The alternative shard list is sorted by priority, and the priority rule is: H3 shard > H2 shard > H1 shard.

4. A data security union processing method according to claim 1, characterized in that, For the local redirection, in the merging stage, only send a directed query request to the shards in the alternative shard list for the conflicting primary key, which specifically includes the following steps: Write the potential conflicting primary key and its alternative shard list into a lightweight conflict table. The key is the byte-type long value of the primary key, and the value is a JSON-formatted list of alternative shard numbers. The lightweight conflict table is implemented based on Redis Cluster, and the conflict table TTL is set to 2 times the merging task timeout time; In the merging stage, if the primary shard matching fails, query the conflict table to obtain the alternative shard list; Send a batch query request to the alternative shards, and each single request contains at most 100 primary keys; If no result is obtained from the alternative shard query, discard the primary key data and record it in the exception log.

5. A data security union processing method according to claim 1, characterized in that For the parallel merge, multi-threaded parallel merge is initiated for the primary shard data, and the conflict query results are incrementally aggregated, specifically including the following steps: Split the merge task of a single shard into multiple sub-tasks. Each sub-task processes a continuous primary key range, and each sub-task starts an independent thread to implement task allocation through a lock-free queue; Write the merge result of the primary shard into Redis Sorted Set according to the shard number. The Score is the long value of the primary key, and the Value is the merged data record; Append the conflict query results to the corresponding Sorted Set in batches through Redis Pipeline; After the merge is completed, export the Sorted Set data in the order of the primary key to generate the final data set.

6. A data security union processing method according to claim 5, characterized in that, For the parallel merge, multi-threaded parallel merge is initiated for the primary shard data, and the conflict query results are incrementally aggregated. It also includes the following steps At the moment when the merge is completed, generate snapshots of the shard data based on the Chandy-Lamport algorithm; Compare the snapshot data of the primary shard and the standby shard to identify the unmatched primary keys; Trigger a secondary merge process for the missing primary keys, limit the maximum number of retries to 3 times. If the secondary merge still fails, mark the primary key as "data that cannot be merged" and push it to the manual review queue.

7. A data security union processing method according to claim 1, characterized in that For the load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters, specifically including the following steps: Collect the data volume, query latency, and conflict rate metrics of the shards; If the data volume of a shard exceeds the preset threshold or the conflict rate > 5%, trigger shard splitting or migration. Split it into two sub-shards, and the sub-shard rules follow the original multi-level hash chain. During the migration process, the old and new shards serve query requests in parallel, and ensure data consistency through the double-write mechanism. After the migration is completed, update the distributed routing table and clean up the old shard data.

8. A data security union processing method according to claim 7, characterized in that, For the load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters. It also includes the following steps: Dynamically adjust the priority of the hash function according to the historical conflict rate. If the conflict rate of H1 exceeds 10% continuously for 3 times, automatically set H2 as the main hash function; Periodically optimize the hash seed, and use the simulated annealing algorithm to find the optimal seed value.

9. A data security union processing method according to claim 8, characterized in that For the load balancing, monitor the shard load in real time, dynamically migrate overloaded shards and adjust the hash parameters. It also includes the following steps: The number of shards is dynamically adjusted. Based on the total data volume D and the upper limit C of the single shard capacity, the number of shards is dynamically calculated according to the formula ; If the current number of shards < N, trigger the shard expansion process, and re-allocate the data according to the multi-level hash chain rules; Conflict rate feedback control, establish a PID controller, with the input being the real-time conflict rate and the output being the hash seed adjustment amount. If the conflict rate exceeds the set threshold, update the hash function parameters through the gradient descent algorithm.

10. A data security union processing system, with operation instructions for performing the data security union processing method as described in claims 1 to 9 when executed, characterized in that, It includes a storage module, a processing module, and an interface module. The interface module and the processing module are connected to the storage module through a bus. The storage module is a memory, the processing module is a processor, and the interface module is a communication interface.

Images