Processing Method and System for Improving the Efficiency of Cross-System Data Consistency Comparison

Abstract

This application relates to the technical field of data processing, and discloses a processing method and system for improving the efficiency of cross-system data consistency comparison, including: performing a data alignment process, including generating a data fingerprint for each data based on consistent hashing; real-time monitoring of data change events, and issuing an analysis and comparison command when a data change event is detected; performing a cross-system data comparison process when the analysis and comparison command is received, and when the comparison result is consistent, the data synchronization is completed, otherwise an alarm is triggered and a repair strategy is executed; adjusting the comparison strategy of the cross-system data comparison process according to the comparison result feedback. The present invention can adjust the cross-system data comparison process through the comparison result feedback. This stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, the comparison strategy adopted in the comparison process can be adjusted in a timely manner, thereby improving the accuracy of the comparison.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a processing method and system for improving the efficiency of cross-system data consistency comparison. Background Art

[0002] During the process of cross-system data synchronization, it is necessary to compare the data of different systems. In the prior art, cross-system data comparison usually adopts a timing synchronization or event-triggered mechanism. The system will regularly check whether the preset data synchronization trigger conditions are met. These conditions are based on time (for example, synchronize once per hour) or based on events (for example, when the database reaches a certain amount of data changes). If the trigger conditions are met, the system will perform a data synchronization operation. If the current trigger conditions are not met, the system will not perform data synchronization but wait until the next check cycle to evaluate the trigger conditions again.

[0003] The existing data synchronization trigger conditions rely on the timing synchronization or event-triggered mechanism in scenarios with large amounts of data or frequent interactions between systems, resulting in delays in data synchronization, inability to reflect data changes in real time, and insufficient protection of data privacy. There are problems such as high latency, low efficiency, and poor privacy. Therefore, under the data federated learning framework, how to improve the cross-system data alignment efficiency while ensuring data privacy is the fundamental problem to be solved by the present invention. Summary of the Invention

[0004] In order to improve the cross-system data alignment efficiency while ensuring data privacy, this application provides a processing method and system for improving the efficiency of cross-system data consistency comparison.

[0005] In a first aspect, this application provides a processing method for improving the efficiency of cross-system data consistency comparison, adopting the following technical solution:

[0006] A processing method for improving the efficiency of cross-system data consistency comparison includes:

[0007] Performing a data alignment process, including generating a data fingerprint for each data based on consistent hashing;

[0008] Real-time monitoring of data change events, and issuing an analysis and comparison command when a data change event is detected;

[0009] Performing a cross-system data comparison process when receiving the analysis and comparison command. When the comparison result is consistent, the data synchronization is completed. Otherwise, an alarm is triggered and a repair strategy is executed;

[0010] Adjusting the comparison strategy for the cross-system data comparison process according to the feedback of the comparison result.

[0011] By adopting the above technical solution, the system generates a data fingerprint for each newly generated data item. When the data changes, the system only needs to recalculate the fingerprints of the affected data and compare them with the fingerprints in the target system, so as to quickly locate the data differences, and then monitor the data change events in real time. By adjusting the cross-system data comparison process through the feedback of the comparison results, this stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, it can timely adjust the comparison strategy adopted in the comparison process, thereby improving the accuracy of the comparison and enhancing the privacy of the data.

[0012] Optionally, the data alignment process further includes adding noise to each data based on differential privacy technology;

[0013] Construct a sample set partitioning strategy based on k-anonymity technology to ensure that each group of data contains at least k individuals.

[0014] By adopting the above technical solution, it is possible to prevent the individual information from being deduced through the data intersection, so that the comparison result cannot be associated with a single user, and the data security and privacy in the data comparison process are improved.

[0015] Optionally, the comparison strategies for the cross-system data comparison process include a first comparison strategy, a second comparison strategy, and a third comparison strategy;

[0016] The first comparison strategy uses data fingerprint comparison;

[0017] The second comparison strategy uses data fingerprint plus incremental sharding comparison;

[0018] The third comparison strategy uses data fingerprint plus full-volume sharding comparison;

[0019] Set the first comparison strategy as the initial comparison strategy. When the comparison result is consistent, keep the initial comparison strategy unchanged. When the comparison result is inconsistent, calculate the comparison coincidence value according to the comparison result, and adjust the initial comparison strategy to one of the second comparison strategy and the third comparison strategy according to the size of the comparison coincidence value.

[0020] By adopting the above technical solution, first, the integrity of the content is compared by calculating the consistent hash value. This process can efficiently implement the data comparison process. When the comparison result is consistent, it indicates that the data across systems is relatively unified, and the initial comparison strategy remains unchanged. When the comparison result is inconsistent, the second and third comparison strategies are adaptively selected according to the degree of inconsistency. In this process, the comparison overlap value is calculated based on the comparison result, and the initial comparison strategy is adjusted to one of the second and third comparison strategies according to the size of the comparison overlap value. This method can adaptively adjust the comparison strategy according to the comparison result, thereby adaptively adjusting the comparison efficiency, achieving the real-time comparison efficiency of improving the cross-system data consistency while meeting the comparison accuracy rate, and reducing the communication overhead at the same time.

[0021] Optionally, the first comparison strategy determines the comparison result through the data fingerprint consistency ratio;

[0022] The second comparison strategy determines the comparison result through the data fingerprint consistency ratio and column-by-column comparison of character segments. The comparison overlap value is the weighted average of the data fingerprint consistency ratio and the column-by-column comparison overlap ratio;

[0023] The third comparison strategy determines the comparison result through the data fingerprint consistency ratio and row-by-row comparison of character segments. The comparison overlap value is the weighted average of the data fingerprint consistency ratio and the row-by-row comparison overlap ratio.

[0024] By adopting the above technical solution, three comparison strategies are provided, and there are differences in the emphasis on comparison efficiency and comparison accuracy among the three comparison strategies. The first comparison strategy uses data fingerprint comparison, that is, the integrity of the content is compared by calculating the consistent hash value. Therefore, the first comparison strategy can reduce the data transmission volume and improve the efficiency of the data comparison process. The second comparison strategy uses data fingerprint plus incremental sharding comparison. On the basis of the first comparison strategy, the second comparison strategy can perform a relatively accurate comparison process on the incremental data by sharding and comparing the incremental data. The third comparison strategy uses data fingerprint plus full-scale sharding comparison. Compared with the comparison of incremental data, the third comparison strategy can perform a relatively accurate comparison process on all data. Therefore, one of them can be adaptively selected according to the comparison result.

[0025] Optionally, when executing the repair strategy, the shard size is dynamically adjusted according to the running data during the execution process of the repair strategy;

[0026] The running data includes the CPU usage rate change curve, the memory usage rate change curve, the single-shard processing delay, and the single-shard data volume size.

[0027] By adopting the above technical solution to dynamically adjust the shard size, the efficiency of the comparison process can be improved. Moreover, the basis for dynamic adjustment is not only based on the single-shard processing latency and the size of single-shard data, but also adjusted according to the CPU usage rate change curve and the memory usage rate change curve. When the CPU usage rate or the memory usage rate occupies abnormally, the processing efficiency can be improved by reducing the shard size.

[0028] Optionally, the process of the dynamic adjustment includes:

[0029] Shard the data according to a preset sharding strategy;

[0030] Calculate the environment trigger value according to the CPU usage rate change curve and the memory usage rate change curve;

[0031] Obtain the trigger value for each shard according to the single-shard processing latency and the size of single-shard data;

[0032] Taking the system's maximum concurrent thread number as a constraint, determine whether each shard is split into smaller shards according to the trigger value of each shard and the environment trigger value.

[0033] By adopting the above technical solution, it can not only meet the requirements of the system's concurrent thread number, but also shard according to the current system's operating environment and the data adaptability in the shard processing process, thereby improving the efficiency of the data comparison process.

[0034] Optionally, the environment trigger value is positively correlated with the CPU usage risk value and the memory usage risk value respectively;

[0035] Current The CPU usage risk value at the time point Pass Calculated; Is the CPU usage rate change curve, Is the first adjustment factor. When the average change rate of the CPU usage rate from the task start time point To the current time point Is less than the preset value At this time, the first adjustment factor ; Otherwise, the first adjustment factor ; where m() is the function to take the maximum value, Is Is The maximum value of the CPU usage rate change rate during the 、 Is the first proportion coefficient;

[0036] Current The memory usage risk value at the time point Pass Calculated; is the memory usage change curve, is the second adjustment factor. When the starting time point of the task to the current time point the average change rate of memory usage is less than the preset value the second adjustment factor ; otherwise, the second adjustment factor , is the maximum value of the memory usage change rate during the , is the second proportion coefficient.

[0037] By adopting the above technical solution, the obtained CPU usage risk value and memory usage risk value can combine factors such as the CPU usage rate, memory usage rate, CPU usage rate change rate status, and memory usage rate change rate to judge the system operation status, and then provide a relatively accurate basis for whether to perform sharding.

[0038] Optionally, the process of obtaining each shard trigger value includes:

[0039] Obtain the delay difference when the single-shard processing delay exceeds the standard delay, and calculate the proportion of the delay difference relative to the delay standard ;

[0040] Calculate the multiple of the single-shard data volume size relative to the average value of all shard data volume sizes ;

[0041] According to and adjust the size of the shard trigger value in a positive correlation.

[0042] Optionally, the process of determining whether each shard is split into smaller shards includes:

[0043] Compare the environment trigger value with the preset trigger threshold:

[0044] If the environment trigger value does not exceed the preset trigger threshold, do not split each shard into smaller shards;

[0045] If the environment trigger value exceeds the preset trigger threshold, sort all shards in descending order according to the trigger value, and split the shards in turn according to the sorting result until the number of threads reaches the system maximum concurrent thread number.

[0046] By adopting the above technical solution, the proportion of the obtained delay difference relative to the delay standard and the multiple of the single-shard data volume size relative to the average value of all shard data volume sizes , when the proportion of the delay difference relative to the delay standard and the multiple of the single-shard data volume size relative to the average value of all shard data volume sizes is larger, it is more necessary to split the shards. Therefore, according to and positively correlate to adjust the shard trigger value size, so that the trigger value of each obtained shard can reflect the demand state of whether the current shard needs to be split.

[0047] In a second aspect, the present application provides a processing system for improving the efficiency of cross-system data consistency comparison, adopting the following technical solutions:

[0048] A processing system for improving the efficiency of cross-system data consistency comparison, the system adopts the processing method for improving the efficiency of cross-system data consistency comparison described in any one of the above.

[0049] Through the above technical solutions, it is possible to adjust the cross-system data comparison process through the comparison result feedback. This stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, it is possible to timely adjust the comparison strategy adopted in the comparison process, thereby improving the accuracy of the comparison.

[0050] In summary, the present application includes at least one of the following beneficial technical effects:

[0051] First, the present invention adjusts the cross-system data comparison process through the comparison result feedback. This stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, it is possible to timely adjust the comparison strategy adopted in the comparison process. Since the present invention sets multiple comparison strategies, and different comparison strategies have different emphases on different efficiencies and comparison accuracies, it is possible to adaptively adjust the comparison strategy according to the comparison result, and then adaptively adjust the comparison efficiency, so as to improve the real-time comparison efficiency of cross-system data consistency on the premise of meeting the comparison accuracy rate, and at the same time improve the privacy of the data.

[0052] Second, the present invention can improve the efficiency of the comparison process by dynamically adjusting the shard size. At the same time, the basis for dynamic adjustment is not only based on the single-shard processing delay and the single-shard data volume size, but also adjusted according to the CPU usage rate change curve and the memory usage rate change curve. When the CPU usage rate or the memory usage rate occupies abnormally, it is possible to improve the processing efficiency by reducing the shard size; through the calculated CPU usage risk value and memory usage risk value, it is possible to combine factors such as the CPU usage rate size, the memory usage rate size, the CPU usage rate change rate state, and the memory usage rate change rate to realize the judgment of the system operation state, and then provide a relatively accurate basis for whether to perform shard splitting. Description of the Drawings

[0053] Figure 1 It is a flowchart of the steps of the processing method for improving the efficiency of cross-system data consistency comparison in the present invention. Specific implementation manners

[0054] The following details the implementation manners of the present application, and examples of the implementation manners are shown in the accompanying drawings.

[0055] In the description of this specification, the description referring to the terms "certain implementation manners", "one implementation manner", "some implementation manners", "schematic implementation manners", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the implementation manner or example are included in at least one implementation manner or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same implementation manner or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more implementation manners or examples.

[0056] The embodiments of the present application disclose a processing method for improving the efficiency of cross-system data consistency comparison. Refer to Figure 1, including: First, a data alignment process is carried out. This process includes generating a data fingerprint for each piece of data based on consistent hashing. In this embodiment, the data fingerprint is generated by encoding the key identifier using a collision-resistant hash function such as SHA-3. Therefore, the data fingerprint is a unique identifier for the data and can also reduce the risk of leakage. In the data comparison process, the system generates a data fingerprint for each newly generated data item and stores it in the fingerprint database. The generation process of the data fingerprint is irreversible, that is, the original data cannot be recovered from the fingerprint, making the data fingerprint an efficient and secure data identification means. In addition, during the data fingerprint generation process, sensitive fields are desensitized. The reflection mechanism is used to automatically identify sensitive fields in the data object (such as ID card numbers, location information), and the processor chain is called to complete the desensitization, thereby ensuring the privacy and security of the data. When the data changes, the system only needs to recalculate the fingerprint of the affected data and compare it with the fingerprint in the target system to quickly locate the data difference. Then, data change events are monitored in real time, and when a data change event is detected, an analysis and comparison command is issued. To ensure the real-time consistency of the data, a real-time data monitoring agent is deployed in the system of this embodiment. The monitoring agent can capture data change events in the source system in real time, such as adding, modifying, or deleting data items. Once a data change is detected, the monitoring agent immediately sends these change events to the data comparison service for processing. Through real-time data monitoring, the system can timely discover and handle data inconsistency problems, thereby avoiding data conflicts and error accumulation. When the system receives the analysis and comparison command, a cross-system data comparison process is carried out. When the comparison result is consistent, the data synchronization is completed; otherwise, an alarm is triggered and a repair strategy is executed. By triggering an alarm to notify relevant personnel, the alarm information includes the detailed information of the inconsistent data, possible reasons, and recommended solutions. At the same time, the system also performs repair operations according to the preset repair strategy to eliminate the data inconsistency problem. The repair operations may include data synchronization, data merging, or data overwriting, etc.; during the repair process, the system ensures the integrity and accuracy of the data and avoids affecting the normal operation of the business system. Finally, the comparison strategy for the cross-system data comparison process is adjusted according to the comparison result feedback. In the above process, this embodiment can adjust the cross-system data comparison process through the comparison result feedback. This stepped comparison method can greatly improve the efficiency of the comparison process. At the same time, when there is a data volume inconsistency, the comparison strategy adopted in the comparison process can be adjusted in a timely manner, thereby improving the accuracy of the comparison. At the same time, through the data fingerprint generation process, the privacy of the data is improved.

[0057] In addition, the data alignment process further includes adding noise to each piece of data based on differential privacy technology. For example, before data comparison, Laplace noise is added to count-type data, or Gaussian noise is used to protect the aggregation result. By introducing noise, it is possible to prevent individual information from being deduced through the data intersection, thereby ensuring the privacy of the data. At the same time, in this embodiment, a sample set partitioning strategy is constructed based on the K-anonymity technology to ensure that each group of data contains at least K individuals, so that the comparison result cannot be associated with a single user, further improving the data security and privacy in the data comparison process.

[0058] In one embodiment, three comparison strategies are set for the cross-system data comparison process, including the first comparison strategy, the second comparison strategy, and the third comparison strategy respectively. The first comparison strategy uses data fingerprint comparison, that is, the complete content is compared by calculating the consistent hash value. Therefore, the first comparison strategy can reduce the data transmission volume and improve the efficiency of the data comparison process. The second comparison strategy uses data fingerprint plus incremental shard comparison. On the basis of the first comparison strategy, the second comparison strategy can perform a relatively accurate comparison process on the incremental data by comparing the incremental shards. The third comparison strategy uses data fingerprint plus full-scale shard comparison. Compared with the comparison of incremental data, the third comparison strategy can perform a relatively accurate comparison process on all the data. The first comparison strategy is set as the initial comparison strategy, that is, first, the complete content is compared by calculating the consistent hash value. This process can efficiently implement the data comparison process. Therefore, when the comparison result is consistent, it indicates that the cross-system data is relatively unified, and the initial comparison strategy remains unchanged. When the comparison result is inconsistent, the second comparison strategy and the third comparison strategy are adaptively selected according to the degree of inconsistency. In this process, the comparison coincidence value is calculated according to the comparison result, and the initial comparison strategy is adjusted to one of the second comparison strategy and the third comparison strategy according to the size of the comparison coincidence value. Through the above process, the comparison strategy can be adaptively adjusted according to the comparison result, and then the comparison efficiency can be adaptively adjusted to improve the real-time comparison efficiency of cross-system data consistency under the premise of meeting the comparison accuracy rate, while also reducing the communication overhead.

[0059] In one embodiment, the first comparison strategy determines the comparison result based on the data fingerprint consistency ratio. That is, when the data fingerprint consistency ratio is higher than the preset ratio standard, the comparison result is determined to be consistent; otherwise, it is determined to be inconsistent. For the second comparison strategy, it determines the comparison result through the data fingerprint consistency ratio and column-by-column comparison of character segments. The comparison coincidence value is the weighted average of the data fingerprint consistency ratio and the column-by-column comparison coincidence ratio. That is, the comparison coincidence value at this time combines the results of the hash consistency comparison and the column-by-column comparison coincidence ratio for comprehensive judgment. When one or both of them are lower than the set standard, or when the two values are too low as a whole, it can reflect that there is an inconsistency problem in the data that requires data synchronization. In addition, the third comparison strategy determines the comparison result through the data fingerprint consistency ratio and row-by-row comparison of character segments. The comparison coincidence value is the weighted average of the data fingerprint consistency ratio and the row-by-row comparison coincidence ratio. That is, the comparison coincidence value at this time combines the results of the hash consistency comparison and the row-by-row comparison coincidence ratio for comprehensive judgment. When one or both of them are lower than the set standard, or when the two values are too low as a whole, it can reflect that there is an inconsistency problem in the data that requires data synchronization.

[0060] In one embodiment, when executing the repair strategy, the shard size is dynamically adjusted according to the running data during the execution process of the repair strategy; the running data includes the CPU usage change curve, the memory usage change curve, the single shard processing delay, and the single shard data volume size. By dynamically adjusting the shard size in the above process, the efficiency of the comparison process can be improved. At the same time, the basis for dynamic adjustment is not only based on the single shard processing delay and the single shard data volume size, but also adjusted according to the CPU usage change curve and the memory usage change curve. When the CPU usage or memory usage occupancy is abnormal, the processing efficiency can be improved by reducing the shard size.

[0061] The process of dynamic adjustment includes: sharding the data according to the preset sharding strategy, where the preset sharding strategy can be selected and combined according to different application fields by combining consistent hashing and range splitting, etc. Then, the environment trigger value is calculated based on the CPU usage change curve and the memory usage change curve, and the trigger value for each shard is obtained according to the single shard processing delay and the single shard data volume size; taking the system's maximum concurrent thread number as a constraint, it is determined whether each shard is split into smaller shards according to the trigger value for each shard and the environment trigger value. In this way, it can not only meet the requirements of the system's concurrent thread number, but also shard according to the current system's running environment and the data adaptability during the shard processing process, thereby improving the efficiency of the data comparison process.

[0062] In one embodiment, the environment trigger value is positively correlated with the CPU usage risk value and the memory usage risk value respectively, where the CPU usage risk value at the current Obtained by calculation; is the CPU usage rate change curve, is the first adjustment factor. When the average change rate of the CPU usage rate from the task start time point to the current time point is less than the preset value At this time, the preset value is set according to the average value in the empirical data. Therefore, when it is less than the preset value, it indicates that the change situation of the CPU usage rate is within the normal range. At this time, the influence of the CPU usage rate change situation is not considered. Therefore, let the first adjustment factor ; Otherwise, the first adjustment factor where m() is the maximum value function. Therefore is is the maximum value of the CPU usage rate change rate during the , is the first proportion coefficient, which is obtained by proportional distribution according to the maximum value of the CPU usage rate change rate and the influence degree of the average change rate of the CPU usage rate in the test data. Therefore, the calculated CPU usage risk value is not only affected by the CPU usage rate at the current time point, but also affected by the CPU usage rate change rate. The greater the current CPU usage rate and the higher the degree to which the CPU usage rate change rate exceeds the average level, the greater the corresponding CPU usage operation risk, that is, the greater the obtained CPU usage risk value; Similarly, the current memory usage risk value at the time point is obtained by calculation; is the memory usage rate change curve, is the second adjustment factor. When the average change rate of the memory usage rate from the task start time point to the current time point is less than the preset value At this time, the second adjustment factor ; Otherwise, the adjustment factor ; , is the maximum value of the memory usage rate change rate during the , is the second proportion coefficient, which is obtained by proportional distribution according to the maximum value of the memory usage rate change rate and the influence degree of the average memory usage rate change rate in the test data. Therefore, the calculated memory usage risk value is affected not only by the memory usage rate at the current time point, but also by the memory usage rate change rate. The greater the current memory usage rate and the higher the degree to which the memory usage rate change rate exceeds the average level, the greater the corresponding memory usage operation risk, that is, the greater the obtained memory usage risk value. Therefore, through the above process of obtaining the environmental trigger value, an accurate reference basis can be provided for whether to perform sharding.

[0063] In one embodiment, the process of obtaining each shard trigger value includes: obtaining the delay difference when the single-shard processing delay exceeds the standard delay, and calculating the proportion of the delay difference relative to the delay standard. Calculating the multiple of the single-shard data volume size relative to the average value of the data volume sizes of all shards. Obviously, when the proportion of the delay difference relative to the delay standard and the multiple of the single-shard data volume size relative to the average value of the data volume sizes of all shards are larger, it is more necessary to split the shards. Therefore, according to and positively adjust the size of the shard trigger value. Furthermore, the obtained trigger value for each shard can reflect the demand status of whether the current shard needs to be split. It should be noted that the specific calculation process of the trigger value can be adaptively adjusted according to the requirements of different fields and The degree of influence is not specifically limited.

[0064] In one embodiment, the process of determining whether each shard is split into smaller shards includes: comparing the environmental trigger value with a preset trigger threshold, which is set according to the data under the critical state of the current system operation. If the environmental trigger value does not exceed the preset trigger threshold, it means that the current system operation state is normal and each shard is not split into smaller shards; if the environmental trigger value exceeds the preset trigger threshold, then sort all shards in descending order according to their trigger values, and split the shards in turn according to the sorting result until the number of threads reaches the maximum concurrent thread number of the system. Through the above process, the adjustment process of the shard size can be adaptively achieved on the premise of meeting the system condition requirements, thereby improving the efficiency of the comparison process.

[0065] The embodiment of the present application also discloses a processing system for improving the efficiency of cross-system data consistency comparison. This system can execute the above-mentioned processing method for improving the efficiency of cross-system data consistency comparison, adjust the cross-system data comparison process through the feedback of the comparison result, greatly improve the efficiency of the comparison process, and at the same time, when there is a data volume inconsistency, it can timely adjust the comparison strategy adopted in the comparison process, thereby improving the accuracy of the comparison.

[0066] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A processing method for improving the efficiency of cross-system data consistency comparison, characterized in that, Including: Perform a data alignment process, including generating a data fingerprint for each data based on consistent hashing; Real-time monitor data change events and issue an analysis and comparison command when a data change event is detected; Perform a cross-system data comparison process when receiving the analysis and comparison command. When the comparison result is consistent, the data synchronization is completed. Otherwise, trigger an alarm and execute a repair strategy; Adjust the comparison strategy of the cross-system data comparison process according to the feedback of the comparison result; The comparison strategy of the cross-system data comparison process includes a first comparison strategy, a second comparison strategy, and a third comparison strategy; The first comparison strategy uses data fingerprint comparison; The second comparison strategy uses data fingerprint plus incremental sharding comparison; The third comparison strategy uses data fingerprint plus full-scale sharding comparison; Set the first comparison strategy as the initial comparison strategy. When the comparison result is consistent, keep the initial comparison strategy unchanged. When the comparison result is inconsistent, calculate the comparison coincidence value according to the comparison result, and adjust the initial comparison strategy to one of the second comparison strategy and the third comparison strategy according to the size of the comparison coincidence value; When executing the repair strategy, dynamically adjust the shard size according to the running data during the execution process of the repair strategy; The running data includes the CPU usage rate change curve, the memory usage rate change curve, the single-shard processing delay, and the single-shard data volume size; The process of the dynamic adjustment includes: Shard the data according to a preset sharding strategy; Calculate the environment trigger value according to the CPU usage rate change curve and the memory usage rate change curve; Obtain each shard trigger value according to the single-shard processing delay and the single-shard data volume size; Taking the system maximum concurrent thread number as a constraint, determine whether each shard is split into smaller shards according to each shard trigger value and the environment trigger value; The environment trigger value is positively correlated with the CPU usage risk value and the memory usage risk value respectively; Current CPU usage risk value at the current time point Obtained by Calculation; is the CPU usage rate change curve, is the first adjustment factor. When the task start time point to the current time point the average change rate of CPU usage rate is less than the preset value the first adjustment factor ; otherwise, the first adjustment factor , where m() is the maximum value function, is the maximum value of the CPU usage rate change rate during the period, , is the first proportion coefficient; Current The memory usage risk value at the current time point is obtained by calculation; is the memory usage rate change curve, is the second adjustment factor. When the average memory usage rate change rate from the task start time point to the current time point is less than the preset value , the second adjustment factor ; otherwise, the second adjustment factor . , is the maximum value of the memory usage rate change rate during the time period, , is the second proportion coefficient; The process of obtaining each shard trigger value includes: Obtain the delay difference where the single - shard processing delay exceeds the standard delay, and calculate the proportion of the delay difference relative to the delay standard ; Calculate the multiple of the size of a single shard's data volume relative to the average of the data volume sizes of all shards ; According to and adjust the sharding trigger value size in positive correlation; The process of determining whether each shard is split into smaller shards includes: Compare the environment trigger value with a preset trigger threshold: If the environment trigger value does not exceed the preset trigger threshold, do not split each shard into smaller shards; If the environment trigger value exceeds the preset trigger threshold, sort all shards in descending order according to the trigger value, and split the shards in turn according to the sorting result until the number of threads reaches the system maximum concurrent thread number.

2. The processing method for improving the efficiency of cross-system data consistency comparison according to claim 1, characterized in that The data alignment process further includes adding noise to each data based on differential privacy technology; Construct a sample set partitioning strategy based on K-anonymity technology to ensure that each group of data contains at least K individuals.

3. The processing method for improving the efficiency of cross-system data consistency comparison according to claim 2, wherein The first comparison strategy judges the comparison result through the data fingerprint consistency ratio; The second comparison strategy judges the comparison result through the data fingerprint consistency ratio and column-by-column comparison of character segments. The comparison coincidence value is the weighted mean of the data fingerprint consistency ratio and the column-by-column comparison coincidence ratio; The third comparison strategy judges the comparison result through the data fingerprint consistency ratio and row-by-row comparison of character segments. The comparison coincidence value is the weighted mean of the data fingerprint consistency ratio and the row-by-row comparison coincidence ratio.

4. A processing system for improving the efficiency of cross-system data consistency comparison, characterized in that, The system adopts the processing method for improving the cross-system data consistency comparison efficiency as described in any one of claims 1-3.

Images