A data comparison system and method

Abstract

This application provides a data comparison system and method. A CPU acquires first and second data; identifies a target algorithm and target sampling ratio using a data statistical model; the CPU samples the first and second data according to the target sampling ratio to obtain third and fourth sampled data; the CPU performs parallel or serial calculations on the third and fourth sampled data using multiple matching algorithms, and sends the third and fourth sampled data to a hash chip; the hash chip performs hash calculations on the third and fourth sampled data; the hash calculation results are sent to the CPU; the CPU receives the hash calculation results of the third and fourth sampled data, as well as the parallel or serial calculation results, and compares the results; based on the comparison results, it is determined whether the first and second data are the same. The solution of this application can improve the efficiency of comparison.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a data comparison system and method. Background Technology

[0002] Currently, data comparison / verification is frequently required in embedded systems, terminals, servers, and networks. For example, before and after data transmission, the sent and received data need to be compared for consistency to determine if retransmission is necessary. However, current data comparison methods often require comparing each digit of the two sets of data sequentially, resulting in relatively low efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a data comparison system and method to solve the problems of low efficiency and high latency in data comparison. The specific technical solution is as follows:

[0004] A first aspect of the embodiments of this application provides a data comparison system, the system comprising: a hash chip and a central processing unit (CPU); the hash chip and the CPU are electrically connected.

[0005] The CPU is used to acquire first and second data to be compared; and to identify the target algorithms and target sampling ratios corresponding to the first and second data through a pre-created data statistical model. The target algorithms include hash algorithms and multiple matching algorithms from a pre-defined set of algorithms. The data statistical model analyzes and automatically optimizes based on historical data comparison records, which include one or more factors among the difference detection capability, computational latency, computational load, computational complexity, and computational power consumption of various algorithms. The data statistical model is used to dynamically filter multiple target algorithms based on these factors and determine the execution order of the multiple target algorithms when they are computed in parallel or serially. It is also used to adjust the target sampling ratio based on the dynamic relationship between historical difference detection capability and computational latency. If the difference detection capability is lower than the expected value, the target sampling ratio is automatically increased; if the difference detection capability is higher than the expected value and the computational latency is greater than the preset latency, the target sampling ratio is automatically decreased. Alternatively, the target sampling ratio is determined based on the pre-defined correspondence between the data's fault tolerance weight and the sampling ratio.

[0006] The CPU is also used to sample the first data according to the target sampling ratio to obtain the third sampled data, and to sample the second data according to the target sampling ratio to obtain the fourth sampled data. The third sampled data and the fourth sampled data are calculated in parallel or serially through the multiple matching algorithms, and the third sampled data and the fourth sampled data are sent to the hash chip.

[0007] The hash chip is used to receive the third sampled data and the fourth sampled data; perform hash calculation on the third sampled data and the fourth sampled data; and send the calculated hash calculation result to the CPU.

[0008] The CPU is also used to receive the hash calculation results of the third sampled data and the hash calculation results of the fourth sampled data, as well as the parallel or serial calculation results, and compare the results; and determine whether the first data and the second data are the same based on the comparison results.

[0009] In one possible implementation, the CPU is specifically used to sort the various matching algorithms in order of increasing computational load, increasing computational latency, increasing computational power consumption, increasing computational complexity, or decreasing difference detection capability to obtain an algorithm sequence; and to perform calculations on the third sampled data and the fourth sampled data respectively according to the order of the various algorithms in the algorithm sequence.

[0010] In one possible implementation, the CPU is further configured to compare the calculation results of the third sampled data and the fourth sampled data corresponding to each algorithm in the algorithm sequence; if they are the same, the third sampled data and the fourth sampled data are sent to the hash chip; if the comparison results corresponding to any algorithm are different, the first data and the second data are determined to be different.

[0011] The CPU is further configured to determine that the first data and the second data are the same if the hash calculation result of the third sampled data and the hash calculation result of the fourth sampled data are the same.

[0012] In one possible implementation, the CPU is further configured to send the first data and the second data to the hash chip if it is determined that the first data and the second data are the same, so that the hash chip receives and performs hash calculation based on the first data and the second data;

[0013] Receive and compare the hash calculation result with the parallel or serial calculation result; determine whether the first data and the second data are the same based on the comparison result.

[0014] In one possible implementation, the CPU is further configured to compare the hash calculation result of the third sampled data with the hash calculation result of the fourth sampled data to obtain a first comparison result; compare the calculation results of the third sampled data and the fourth sampled data corresponding to any matching algorithm to obtain a second comparison result; and determine that the first data and the second data are different when the first comparison result indicates inconsistency or the second comparison result indicates inconsistency.

[0015] In one possible implementation, the CPU is further configured to stop comparing data and send a stop calculation instruction to the hash chip when it is determined that the first data and the second data are different.

[0016] In one possible implementation, the CPU is further configured to receive the accuracy of the feedback comparison result; and to correct the correspondence between the preset fault tolerance weight and the sampling ratio based on the accuracy, wherein, during the correction, if the accuracy is lower than a preset threshold, the multiple preset sampling ratios corresponding to the fault tolerance weight are increased.

[0017] Based on the accuracy, the pre-created data statistical model is corrected, wherein if the accuracy is higher than a preset threshold, the probability of selecting the multiple matching algorithms is increased.

[0018] In one possible implementation, the CPU is further configured to perform parallel or serial calculations on the first data and the second data using the multiple matching algorithms, and compare the calculation results; if the calculation results indicate that they are consistent, the first data and the second data are sent to the hash chip.

[0019] Receive and compare the hash calculation results; determine whether the first data and the second data are the same based on the comparison results.

[0020] A second aspect of this application provides a data comparison method applied to a data comparison system, the system comprising: a hash chip and a central processing unit (CPU); the hash chip and the CPU are electrically connected; the method comprises:

[0021] The CPU acquires first and second data to be compared; through a pre-created data statistical model, it identifies the target algorithm and target sampling ratio corresponding to the first and second data; wherein, the target algorithm includes: a hash algorithm and multiple matching algorithms from a pre-defined set of algorithms; the data statistical model analyzes and automatically optimizes based on historical data comparison records, the historical data comparison records include one or more factors among the difference detection capability, computation latency, computational load, computational complexity, and computational power consumption of various algorithms; the data statistical model is used to dynamically filter multiple target algorithms according to the factors and determine the execution order of the multiple target algorithms when they are computed in parallel or serially; it is also used to adjust the target sampling ratio according to the dynamic relationship between historical difference detection capability and computation latency, automatically increasing the target sampling ratio if the difference detection capability is lower than the expected value, and automatically decreasing the target sampling ratio if the difference detection capability is higher than the expected value and the computation latency is greater than the preset latency, or, determining the target sampling ratio according to the correspondence between the pre-defined fault tolerance weight of the data and the sampling ratio;

[0022] The CPU samples the first data according to the target sampling ratio to obtain the third sampled data, and samples the second data according to the target sampling ratio to obtain the fourth sampled data. The third sampled data and the fourth sampled data are calculated in parallel or serially through the multiple matching algorithms, and the third sampled data and the fourth sampled data are sent to the hash chip.

[0023] The hash chip receives the third and fourth sampled data; performs hash calculations on the third and fourth sampled data; and sends the calculated hash result to the CPU.

[0024] The CPU receives the hash calculation results of the third sampled data and the hash calculation results of the fourth sampled data, as well as the parallel or serial calculation results, and compares the results; based on the comparison results, it determines whether the first data and the second data are the same.

[0025] In one possible implementation, the step of performing parallel or serial calculations on the third and fourth sampled data using the multiple matching algorithms includes:

[0026] The CPU sorts the various matching algorithms in order of increasing computational load, increasing computational latency, increasing computational power consumption, increasing computational complexity, or decreasing difference detection capability to obtain an algorithm sequence. Then, it performs calculations on the third and fourth sampled data according to the order of the algorithms in the algorithm sequence.

[0027] In one possible implementation, determining whether the first data and the second data are the same based on the comparison result includes:

[0028] The CPU compares the calculation results of the third and fourth sampled data corresponding to each algorithm in the algorithm sequence; if they are the same, the third and fourth sampled data are sent to the hash chip; if the comparison results corresponding to any algorithm are different, the first data and the second data are determined to be different.

[0029] If the hash calculation results of the third sampled data and the fourth sampled data are the same, the CPU determines that the first data and the second data are the same.

[0030] In one possible implementation, the method further includes:

[0031] If the CPU determines that the first data and the second data are the same, it sends the first data and the second data to the hash chip so that the hash chip receives them and performs hash calculations based on the first data and the second data.

[0032] Receive and compare the hash calculation result with the parallel or serial calculation result; determine whether the first data and the second data are the same based on the comparison result.

[0033] In one possible implementation, the method further includes:

[0034] The CPU compares the hash calculation results of the third sampled data and the fourth sampled data to obtain a first comparison result; the CPU then compares the calculation results of the third sampled data and the fourth sampled data with any matching algorithm to obtain a second comparison result; when the first comparison result or the second comparison result is inconsistent, it is determined that the first data and the second data are different.

[0035] In one possible implementation, the method further includes:

[0036] When the CPU determines that the first data and the second data are different, it stops comparing the data and sends a stop calculation command to the hash chip.

[0037] In one possible implementation, the method further includes:

[0038] The accuracy of the comparison results received by the CPU is assessed; based on the accuracy, the correspondence between the preset fault tolerance weight and the sampling ratio is corrected, wherein, during the correction, if the accuracy is lower than a preset threshold, the various preset sampling ratios corresponding to the fault tolerance weight are increased.

[0039] Based on the accuracy, the pre-created data statistical model is corrected, wherein if the accuracy is higher than a preset threshold, the probability of selecting the multiple matching algorithms is increased.

[0040] In one possible implementation, the method further includes:

[0041] The CPU uses multiple matching algorithms to calculate the first and second data in parallel or serially, and compares the calculation results. If the calculation results are consistent, the first and second data are sent to the hash chip.

[0042] Receive and compare the hash calculation results; determine whether the first data and the second data are the same based on the comparison results.

[0043] Beneficial effects of the embodiments in this application:

[0044] This application provides a data comparison system and method. The system includes a hash chip and a central processing unit (CPU). The CPU is used to acquire first data and second data to be compared; identify the first data and second data, their corresponding target algorithms and target sampling ratios through a pre-created data statistical model; the CPU is also used to sample the first data according to the target sampling ratio to obtain third sampled data, and sample the second data according to the target sampling ratio to obtain fourth sampled data; perform parallel or serial calculations on the third sampled data and fourth sampled data using multiple matching algorithms, and send the third sampled data and fourth sampled data to the hash chip; the hash chip is used to receive the third sampled data and fourth sampled data; perform hash calculations on the third sampled data and fourth sampled data; send the calculated hash calculation results to the CPU; the CPU is also used to receive the hash calculation results of the third sampled data and fourth sampled data, as well as the parallel or serial calculation results, and compare the results; and determine whether the first data and second data are the same based on the comparison results. According to the scheme of this application embodiment, after the processor receives the first data and the second data to be compared, it identifies the corresponding matching algorithms and target sampling ratios through a pre-created data statistical model, and then sends the sampled data to the hash chip. The hash chip performs hash calculation on the sampled data, and then compares and calculates through multiple algorithms. Since the hash chip is a chip specifically designed for hash calculation, it can improve the calculation efficiency, thereby improving the comparison efficiency and solving the problems of low efficiency and high latency when comparing data. Furthermore, it improves the accuracy of the comparison through multiple algorithms.

[0045] Furthermore, by introducing an intelligent sampling mechanism, the optimal target sampling ratio is dynamically determined by the data statistical model. When processing large-scale data comparisons, the system only needs to perform calculations and hashing on the small sampled dataset, instead of the full data comparison as in traditional methods. This significantly reduces the amount of data processing, lowers the computational load on the CPU and hash chip, and shortens the comparison response time, thereby significantly improving the overall operating efficiency and performance of the system. The system's built-in data statistical model also possesses self-learning and adaptive optimization capabilities. By comprehensively analyzing the multi-dimensional performance indicators of various algorithms in historical data comparison records (such as difference detection capability, computational latency, and power consumption), it dynamically selects the most suitable combination of target algorithms for the current data characteristics and intelligently decides their execution order. Simultaneously, the model can dynamically adjust the sampling strategy based on historical performance, pursuing optimal efficiency while ensuring accuracy, enabling the system to continuously evolve and flexibly cope with different types of data and diverse comparison needs.

[0046] Furthermore, the data statistical model supports multiple strategy configurations, allowing for dynamic adjustment of sampling ratios based on real-time feedback, or static mapping based on pre-defined fault tolerance weights. It enables flexible selection of appropriate comparison strategies based on different emphases on dimensions such as accuracy, efficiency, and power consumption, meeting the personalized needs of various application scenarios and significantly improving the system's practicality and applicability.

[0047] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0049] Figure 1 A schematic diagram of the structure of a data comparison system provided in an embodiment of this application;

[0050] Figure 2 This is a flowchart illustrating a data comparison method according to an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0052] A first aspect of the embodiments of this application provides a data comparison system, see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of a data comparison system provided in an embodiment of this application. The system includes: a hash chip 101 and a central processing unit (CPU) 102; the hash chip and the CPU are electrically connected.

[0053] The CPU 102 is used to acquire first data and second data to be compared; and to identify the target algorithm and target sampling ratio corresponding to the first data and the second data through a pre-created data statistical model. The target algorithm includes a hash algorithm and multiple matching algorithms from a pre-defined set of algorithms. The data statistical model analyzes and automatically optimizes based on historical data comparison records, which include one or more factors among the difference detection capability, computational latency, computational load, computational complexity, and computational power consumption of various algorithms. The data statistical model is used to dynamically filter multiple target algorithms according to the factors and determine the execution order of the multiple target algorithms when they are computed in parallel or serially. It is also used to adjust the target sampling ratio according to the dynamic relationship between historical difference detection capability and computational latency. If the difference detection capability is lower than the expected value, the target sampling ratio is automatically increased; if the difference detection capability is higher than the expected value and the computational latency is greater than the preset latency, the target sampling ratio is automatically decreased. Alternatively, the target sampling ratio is determined according to the pre-defined correspondence between the data's fault tolerance weight and the sampling ratio.

[0054] The CPU 102 is also used to sample the first data according to the target sampling ratio to obtain the third sampled data, and to sample the second data according to the target sampling ratio to obtain the fourth sampled data. The third sampled data and the fourth sampled data are calculated in parallel or serially through the multiple matching algorithms, and the third sampled data and the fourth sampled data are sent to the hash chip.

[0055] The hash chip 101 is used to receive the third sampled data and the fourth sampled data; perform hash calculation on the third sampled data and the fourth sampled data; and send the calculated hash calculation result to the CPU 102.

[0056] The CPU 102 is also used to receive the hash calculation results of the third sampled data and the hash calculation results of the fourth sampled data, as well as the parallel or serial calculation results, and compare them; and determine whether the first data and the second data are the same based on the comparison results.

[0057] Specifically, the data statistical model is the core of the system's intelligent decision-making, possessing self-learning and adaptive optimization capabilities. This data statistical model is not statically configured, but rather continuously analyzed and iteratively optimized based on long-term accumulated historical data comparison records.

[0058] 1. The foundation for learning and optimizing data statistical models.

[0059] Historical data comparison records constitute the training data and experience base for the model's decision-making. Each record contains multi-dimensional performance metrics for a complete data comparison task, including but not limited to:

[0060] Difference detection capability is used to measure the effectiveness of different algorithms in successfully detecting data differences in the past, such as difference detection rate and false detection rate.

[0061] Calculate latency and record the time taken for each algorithm to complete the comparison calculation.

[0062] Computational load and computational complexity reflect the degree to which an algorithm requires computational resources (such as CPU cycles and memory usage).

[0063] Calculate power consumption. Record the energy consumption metrics during the execution of each algorithm.

[0064] By comprehensively analyzing one or more of the above historical factors, the data statistical model can deeply explore the performance characteristics and patterns of different algorithms under different data types and scenarios, thereby laying the foundation for intelligent decision-making.

[0065] 2. Intelligent algorithm strategy decision-making.

[0066] Based on the above analysis, the data statistical model undertakes the following algorithmic decision-making functions:

[0067] (1) Dynamic selection of target algorithms. Based on the characteristics of the first and second data to be compared, the model can dynamically select multiple most suitable target algorithms from the pre-set algorithm library in real time. This selection process comprehensively weighs multiple objectives such as difference detection capability, efficiency (latency, computational load) and power consumption, aiming to configure the optimal algorithm combination for achieving accurate, efficient and low-power comparison tasks.

[0068] (2) Determine the execution order. The data statistical model further plans the execution scheme for the selected target algorithms. This scheme includes determining whether the algorithm adopts parallel computing or serial computing mode. In serial mode, the data statistical model will determine the order of their execution (i.e., execution order), for example, prioritizing the execution of algorithms with small computational load and short time consumption in order to quickly discover differences; or prioritizing the execution of algorithms with the strongest difference detection capability to reduce the risk of missed detection.

[0069] 3. Intelligent sampling strategy decision-making.

[0070] When determining the target sampling proportion, the data statistical model provides the following two configurable strategies:

[0071] Strategy A: Dynamic adaptive adjustment based on feedback.

[0072] The data statistical model continuously monitors the dynamic relationship between historical difference detection capability and computation latency, and automatically adjusts the target sampling ratio for the next comparison accordingly, forming a closed-loop feedback system. Specifically, this can include the following two sampling ratios:

[0073] (1) Increase the sampling ratio: When the historical difference detection capability is consistently lower than the preset expected value, it indicates that the current sampling ratio may be too low, and there is a risk of missed detection. The model will automatically increase the target sampling ratio to improve the accuracy and reliability of the comparison by increasing the amount of data sampled. The target sampling ratio can be 100%. It can be understood that when the target sampling ratio is 100%, it is not necessary to perform sampling operations on the first and second data.

[0074] (2) Reduce the sampling ratio: When the historical difference detection capability is higher than expected, but the calculation delay exceeds the preset allowable time, it indicates that the current sampling ratio may be too high, causing unnecessary resource consumption and time delay. The model will automatically reduce the target sampling ratio, which will significantly improve the comparison efficiency and reduce the system overhead while ensuring sufficient accuracy.

[0075] Strategy B: Static mapping based on fault-tolerant weights.

[0076] Furthermore, the data statistical model also supports a rule-based sampling strategy. The system can pre-define different fault tolerance weights for data of different importance levels (e.g., high fault tolerance weight for core system data and low fault tolerance weight for ordinary log data), and establish a mapping table between fault tolerance weights and sampling ratios (e.g., higher weight data corresponds to a higher sampling ratio or even a full comparison, i.e., the sampling ratio can be 100%, in which case sampling operations are not required for the first and second data). The data statistical model can directly determine the target sampling ratio by querying this mapping table based on the current data's fault tolerance weight. This method provides an effective supplement and safety net for dynamic strategies.

[0077] After a detailed description of the data statistical model provided in the embodiments of the present invention, the data comparison process will be explained in detail below.

[0078] The CPU is used to acquire first and second data to be compared. Specifically, the first and second data are two sets of data to be compared; for example, one is standard data and the other is the data to be compared. For instance, one is a correct checksum, and the other is a received checksum to be compared. A pre-created data statistical model can match multiple target algorithms and target sampling ratios corresponding to the first and second data for comparison. The CPU samples the first and second data according to the target sampling ratios, obtaining third and fourth sampled data in sequence. The method for determining multiple target algorithms and target sampling ratios has been explained in detail in the above description of the data statistical model and will not be repeated here. The target algorithms include: hash algorithms and multiple matching algorithms from a pre-defined set of algorithms. The matching algorithms can be sampling verification algorithms, parity verification algorithms, XOR verification algorithms, checksum verification algorithms, CRC (Cyclic Redundancy Check) verification algorithms, etc. The hash chip performs hash calculations, specifically using its built-in hash algorithm. For example, the received third and fourth sampled data are hashed using algorithms such as MD5 (Message-Digest Algorithm), SHA-1 (Secure Hash Algorithm 1), SHA-256 (256-bit hash algorithm), and CRC32 (32-bit cyclic redundancy check algorithm), and the calculation results are sent to the CPU. The CPU is also used to perform parallel or serial calculations on the third and fourth sampled data using the various matching algorithms, and compare the calculation results. The CPU is also used to receive the hash calculation results of the third and fourth sampled data, as well as the parallel or serial calculation results, and compare these results to determine whether the first and second data are the same. When all three results are consistent, the first and second data are determined to be the same; if any comparison result is different, the first and second data are determined to be different.

[0079] Understandably, when the current sampling rate is 100%, the CPU does not need to perform sampling operations on the first and second data. In other words, in this case, the first data is completely consistent with the third sampled data, and the second data is completely consistent with the fourth sampled data.

[0080] As can be seen, through the solution of this application embodiment, after the processor receives the first data and the second data to be compared, it identifies the corresponding matching algorithms and target sampling ratios through a pre-created data statistical model, and then sends the sampled data to the hash chip. The hash chip performs hash calculation on the sampled data, and thus compares and calculates through multiple algorithms. Since the hash chip is a chip specifically designed for hash calculation, it can improve the calculation efficiency, thereby improving the comparison efficiency and solving the problems of low efficiency and high latency in data comparison. Furthermore, it improves the accuracy of comparison through multiple algorithms.

[0081] In this embodiment, the first data is sampled according to the target sampling ratio to obtain the third sampled data; the second data is sampled according to the target sampling ratio to obtain the fourth sampled data. The sampling ratio in this application embodiment can be 1 / 8, 1 / 32, 1 / 128, etc. In actual use, during sampling, a sampling mode sent by the user can also be received, and sampling can be performed according to that sampling mode. In actual use, the sampling mode can include: fixed sampling, random sampling, weighted sampling combined with a statistical model, and dynamic sampling. Fixed sampling can be performed according to a fixed address offset. Random sampling can select specific bits of data for comparison based on the data length and the required comparison accuracy. Weighted sampling combined with a statistical model can allocate sampling weights based on the statistical model of historical data. For example, a higher sampling ratio is used for scenarios with a high probability of error, and a lower sampling ratio is used for scenarios with a low probability of error. Furthermore, the sampling ratio can be increased (even to full coverage) for data with low tolerance for errors in the comparative analysis results, while the sampling ratio can be appropriately decreased for data with high tolerance for errors in the comparative analysis results. Dynamic sampling can introduce a feedback mechanism for data comparison results: when the accuracy of the current data comparison results is lower than expected, the sampling ratio can be increased; when the accuracy of the current data is significantly higher than expected but efficiency is low, the sampling ratio can be appropriately decreased.

[0082] Furthermore, by introducing an intelligent sampling mechanism, the optimal target sampling ratio is dynamically determined by the data statistical model. When processing large-scale data comparisons, the system only needs to perform calculations and hashing on the small sampled dataset, instead of the full data comparison as in traditional methods. This significantly reduces the amount of data processing, lowers the computational load on the CPU and hash chip, and shortens the comparison response time, thereby significantly improving the overall operating efficiency and performance of the system. The system's built-in data statistical model also possesses self-learning and adaptive optimization capabilities. By comprehensively analyzing the multi-dimensional performance indicators of various algorithms in historical data comparison records (such as difference detection capability, computational latency, and power consumption), it dynamically selects the most suitable combination of target algorithms for the current data characteristics and intelligently decides their execution order. Simultaneously, the model can dynamically adjust the sampling strategy based on historical performance, pursuing optimal efficiency while ensuring accuracy, enabling the system to continuously evolve and flexibly cope with different types of data and diverse comparison needs.

[0083] Furthermore, the data statistical model supports multiple strategy configurations, allowing for dynamic adjustment of sampling ratios based on real-time feedback, or static mapping based on pre-defined fault tolerance weights. It enables flexible selection of appropriate comparison strategies based on different emphases on dimensions such as accuracy, efficiency, and power consumption, meeting the personalized needs of various application scenarios and significantly improving the system's practicality and applicability.

[0084] In one possible implementation, the CPU is specifically used to sort the various matching algorithms in order of increasing computational load, increasing computational latency, increasing computational power consumption, increasing computational complexity, or decreasing difference detection capability to obtain an algorithm sequence; and to perform calculations on the third sampled data and the fourth sampled data respectively according to the order of the various algorithms in the algorithm sequence.

[0085] Of course, in practical application scenarios, the various matching algorithms can also be sorted according to actual needs, in order of computational load from large to small, computational latency from large to small, computational power consumption from large to small, computational complexity from large to small, or difference detection capability from weak to strong, to obtain an algorithm sequence. All of these are reasonable. The embodiments of the present invention do not impose specific limitations on the sorting method of computational load, computational latency, computational power consumption, computational complexity, and difference detection capability.

[0086] In one possible implementation, the CPU is further configured to, if it is determined that the first data and the second data are the same, send the first data and the second data to the hash chip, so that the hash chip receives and performs hash calculation based on the first data and the second data; receive and compare the hash calculation result with the parallel or serial calculation result; and determine whether the first data and the second data are the same based on the comparison result. In one possible implementation, the CPU is further configured to compare the calculation results of the third sampled data and the fourth sampled data corresponding to each algorithm in the algorithm sequence; if they are all the same, send the third sampled data and the fourth sampled data to the hash chip; if the comparison results corresponding to any algorithm are different, determine that the first data and the second data are different; the CPU is further configured to, if the hash calculation result of the third sampled data and the hash calculation result of the fourth sampled data are the same, determine that the first data and the second data are the same. The first data is sampled using the target sampling ratio to obtain the third sampled data; the second data is sampled using the target sampling ratio to obtain the fourth sampled data. The CPU samples the data using various preset sampling ratios and then sorts the various verification methods according to their computational overhead from lowest to highest, selecting the method with the lowest computational overhead and highest verification efficiency for the earliest processing. The sampled data is then sent to the hash chip for calculation.

[0087] In one possible implementation, the CPU is further configured to compare the hash calculation result of the third sampled data with the hash calculation result of the fourth sampled data to obtain a first comparison result; compare the calculation results of the third sampled data and the fourth sampled data corresponding to any matching algorithm to obtain a second comparison result; and determine that the first data and the second data are different when the first comparison result indicates inconsistency or the second comparison result indicates inconsistency. In another possible implementation, the CPU is further configured to stop the data comparison and send a stop calculation instruction to the hash chip when it is determined that the first data and the second data are different. When any comparison result is different, it can be directly determined that they are different, thereby stopping the comparison, stopping the data transmission and calculation, and thus improving the efficiency of the comparison. In another example, the method of this application embodiment can also perform parallel calculations of multiple matching algorithms. When processing in parallel, multiple algorithms are performed in parallel, and after obtaining the calculation result of any parallel algorithm, the calculation result is directly compared to determine whether they are the same. When any comparison result is different, they can be directly determined that they are different. In one example, the parallel verification mode can perform verification in multiple modes. It exits if any verification detects a data difference, and the verification process only ends when all verifications pass, indicating complete data consistency. The number and combination of verification algorithms are not fixed; users can choose and configure them flexibly. In practical use, users can select the most suitable verification scheme based on their needs for accuracy, efficiency, computational overhead, cost, and power consumption. (They can choose only one verification algorithm, or multiple algorithms simultaneously for serial or parallel verification. When selecting multiple algorithms, they can also combine alternative methods, such as a combination of random sampling verification and hash verification.) During intelligent verification, users can match and select verification schemes with corresponding characteristics based on their needs for accuracy, efficiency, and cost. Compared to specific verification modes, this type of mode is suitable for users who are not familiar with the principles and characteristics of various verification methods but are familiar with their verification requirements.

[0088] In one possible implementation, the CPU is further configured to receive the accuracy of the feedback comparison results; based on the accuracy, correct the correspondence between the pre-set fault tolerance weights and sampling ratios, wherein, during correction, if the accuracy is lower than a preset threshold, the multiple preset sampling ratios corresponding to the fault tolerance weights are increased; based on the accuracy, correct the pre-created data statistical model, wherein, if the accuracy is higher than a preset threshold, the probability of selecting the multiple matching algorithms is increased. For example, when user feedback indicates low accuracy, the multiple preset sampling ratios corresponding to the fault tolerance weights for this judgment can be corrected. In actual use, users can also choose the corresponding verification mode or a combination of verification modes. Specifically, users can select verification schemes with corresponding characteristics based on their needs for accuracy, efficiency, cost, etc.; for example, Mode 1 (cost-optimal mode): defaults to parity check; Mode 2 (efficiency-optimal mode): defaults to CRC check; Mode 3 (accuracy-optimal mode): defaults to hash check.

[0089] The method described in this application embodiment enables the device to implement a result notification and synchronization mechanism when data results are obtained. This mechanism involves notifying the CPU of the comparison results of two data segments via a specific interrupt; notifying / synchronizing the data comparison results to the on-chip (on the same chip as the comparison device) circuit unit that is interested in the results via a specific logic circuit (such as dedicated pins, signals, messages, etc.); and notifying / synchronizing the data comparison results to the off-chip (on a different chip than the comparison device) circuit unit via a specific circuit (such as GPIO (General-purpose input / output)).

[0090] A second aspect of this application provides a data comparison method applied to a data comparison system, the system comprising: a hash chip and a central processing unit (CPU); the hash chip and the CPU are electrically connected; see also Figure 2 , Figure 2 This is a flowchart illustrating a data comparison method according to an embodiment of this application. The method includes:

[0091] Step S21: Obtain the first data and the second data to be compared through the CPU; identify the target algorithm corresponding to the first data and the second data through a pre-created data statistical model; wherein, the target algorithm includes: a hash algorithm and multiple matching algorithms from a pre-set set of multiple algorithms; perform parallel or serial calculations on the first data and the second data through the multiple matching algorithms, and send the first data and the second data to the hash chip;

[0092] Step S21: The CPU acquires the first data and the second data to be compared; and identifies the target algorithm and target sampling ratio corresponding to the first data and the second data through a pre-created data statistical model.

[0093] The target algorithm includes a hash algorithm and multiple matching algorithms from a pre-defined set of algorithms. The data statistical model analyzes and automatically optimizes based on historical data comparison records, which include one or more factors such as the difference detection capability, computation latency, computational load, computational complexity, and computational power consumption of various algorithms. The data statistical model is used to dynamically select multiple target algorithms based on the factors and determine the execution order of the multiple target algorithms when they are computed in parallel or serially. It is also used to adjust the target sampling ratio based on the dynamic relationship between historical difference detection capability and computation latency. If the difference detection capability is lower than the expected value, the target sampling ratio is automatically increased. If the difference detection capability is higher than the expected value and the computation latency is greater than the preset latency, the target sampling ratio is automatically decreased. Alternatively, the target sampling ratio is determined based on the correspondence between the pre-defined fault tolerance weight of the data and the sampling ratio.

[0094] In step S22, the CPU samples the first data according to the target sampling ratio to obtain the third sampled data, and samples the second data according to the target sampling ratio to obtain the fourth sampled data. The third sampled data and the fourth sampled data are calculated in parallel or serially using the multiple matching algorithms, and the third sampled data and the fourth sampled data are sent to the hash chip.

[0095] Step S23: The hash chip receives the third sampled data and the fourth sampled data; performs hash calculation on the third sampled data and the fourth sampled data; and sends the calculated hash calculation result to the CPU.

[0096] Step S24: The CPU receives the hash calculation result of the third sampled data and the hash calculation result of the fourth sampled data, as well as the parallel or serial calculation result, and compares the results; based on the comparison result, it determines whether the first data and the second data are the same.

[0097] In one possible implementation, the step of performing parallel or serial calculations on the third and fourth sampled data using the multiple matching algorithms includes:

[0098] The CPU sorts the various matching algorithms in order of increasing computational load, increasing computational latency, increasing computational power consumption, increasing computational complexity, or decreasing difference detection capability to obtain an algorithm sequence. Then, it performs calculations on the third and fourth sampled data according to the order of the algorithms in the algorithm sequence.

[0099] In one possible implementation, determining whether the first data and the second data are the same based on the comparison result includes:

[0100] The CPU compares the calculation results of the third and fourth sampled data corresponding to each algorithm in the algorithm sequence; if they are the same, the third and fourth sampled data are sent to the hash chip; if the comparison results corresponding to any algorithm are different, the first data and the second data are determined to be different.

[0101] If the hash calculation results of the third sampled data and the fourth sampled data are the same, the CPU determines that the first data and the second data are the same.

[0102] In one possible implementation, the method further includes:

[0103] If the CPU determines that the first data and the second data are the same, it sends the first data and the second data to the hash chip so that the hash chip receives them and performs hash calculations based on the first data and the second data.

[0104] Receive and compare the hash calculation result with the parallel or serial calculation result; determine whether the first data and the second data are the same based on the comparison result.

[0105] In one possible implementation, the method further includes:

[0106] The CPU compares the hash calculation results of the third sampled data and the fourth sampled data to obtain a first comparison result; the CPU then compares the calculation results of the third sampled data and the fourth sampled data with any matching algorithm to obtain a second comparison result; when the first comparison result or the second comparison result is inconsistent, it is determined that the first data and the second data are different.

[0107] In one possible implementation, the method further includes:

[0108] When the CPU determines that the first data and the second data are different, it stops comparing the data and sends a stop calculation command to the hash chip.

[0109] In one possible implementation, the method further includes:

[0110] The accuracy of the comparison results received by the CPU is assessed; based on the accuracy, the correspondence between the preset fault tolerance weight and the sampling ratio is corrected, wherein, during the correction, if the accuracy is lower than a preset threshold, the various preset sampling ratios corresponding to the fault tolerance weight are increased.

[0111] Based on the accuracy, the pre-created data statistical model is corrected, wherein if the accuracy is higher than a preset threshold, the probability of selecting the multiple matching algorithms is increased.

[0112] In one possible implementation, the method further includes:

[0113] The CPU uses multiple matching algorithms to calculate the first and second data in parallel or serially, and compares the calculation results. If the calculation results are consistent, the first and second data are sent to the hash chip.

[0114] Receive and compare the hash calculation results; determine whether the first data and the second data are the same based on the comparison results.

[0115] As can be seen, through the solution of this application embodiment, after the processor receives the first data and the second data to be compared, it identifies the corresponding matching algorithms through a pre-created data statistical model, and then sends the sampled data to the hash chip. The hash chip performs hash calculation on the sampled data, and thus compares and calculates through multiple algorithms. Since the hash chip is a chip specifically designed for hash calculation, it can improve the calculation efficiency, thereby improving the comparison efficiency and solving the problems of low efficiency and high latency when comparing data. Furthermore, it improves the accuracy of the comparison through multiple algorithms.

[0116] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system, electronic device, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0119] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A data comparison system, characterized by, The system comprises a hash chip and a central processing unit (CPU); the hash chip and the CPU are electrically connected; The CPU is configured to: acquire first data and second data to be compared; identify a target algorithm and a target sampling ratio corresponding to the first data and the second data by using a pre-created data statistical model; the target algorithm comprises a hash algorithm and a plurality of matching algorithms in a plurality of pre-set algorithms; the data statistical model is analyzed and automatically optimized based on historical data comparison records, the historical data comparison records comprise one or more factors in difference detection capability, calculation time delay, calculation amount, calculation complexity and calculation power consumption of various algorithms; the data statistical model is configured to dynamically screen a plurality of target algorithms according to the factors, and determine an execution sequence of the plurality of target algorithms in parallel or serial calculation; and further configured to adjust the target sampling ratio according to a dynamic relationship between historical difference detection capability and calculation time delay, automatically increase the target sampling ratio if the difference detection capability is lower than an expected value, automatically decrease the target sampling ratio if the difference detection capability is higher than the expected value and the calculation time delay is greater than a pre-set time delay, or determine the target sampling ratio according to a pre-set corresponding relationship between a fault tolerance weight of data and a sampling ratio; The CPU is further configured to sample the first data according to the target sampling ratio to obtain third sampling data, sample the second data according to the target sampling ratio to obtain fourth sampling data, and perform parallel or serial calculation on the third sampling data and the fourth sampling data by using the plurality of matching algorithms, and send the third sampling data and the fourth sampling data to the hash chip; The hash chip is configured to receive the third sampling data and the fourth sampling data, perform hash calculation on the third sampling data and the fourth sampling data, and send a hash calculation result obtained by the calculation to the CPU; The CPU is further configured to receive the hash calculation result of the third sampling data and the hash calculation result of the fourth sampling data, and perform parallel or serial calculation on the results, and compare the results; and determine whether the first data and the second data are the same according to a comparison result.

2. The system of claim 1, wherein The CPU is specifically configured to sort the plurality of matching algorithms in an algorithm sequence according to calculation amount from small to large, calculation time delay from small to large, calculation power consumption from small to large, calculation complexity from small to large, or difference detection capability from strong to weak, and perform calculation on the third sampling data and the fourth sampling data according to the order of various algorithms in the algorithm sequence.

3. The system of claim 2, wherein The CPU is further configured to compare calculation results of the third sampling data and the fourth sampling data corresponding to each algorithm in the algorithm sequence; if the calculation results are the same, the third sampling data and the fourth sampling data are sent to the hash chip; if the comparison results corresponding to any algorithm are different, it is determined that the first data and the second data are different. The CPU is further configured to determine that the first data and the second data are identical if the hash calculation result of the third sampling data is identical to the hash calculation result of the fourth sampling data.

4. The system of claim 3, wherein, The CPU is further configured to send the first data and the second data to the hash chip for the hash chip to receive and perform hash calculation based on the first data and the second data if it is determined that the first data and the second data are identical. receive and compare the hash calculation result with the parallel or serial calculation result; and determine whether the first data and the second data are identical based on the comparison result.

5. The system of claim 4, wherein, The CPU is further configured to compare the hash calculation result of the third sampling data with the hash calculation result of the fourth sampling data to obtain a first comparison result, compare the calculation result of the third sampling data and the fourth sampling data corresponding to any one of the plurality of matching algorithms to obtain a second comparison result, and determine that the first data and the second data are different when the first comparison result indicates inconsistency or the second comparison result indicates inconsistency.

6. The system of claim 1, wherein, The CPU is further configured to stop data comparison and send a stop calculation instruction to the hash chip when it is determined that the first data and the second data are different.

7. The system of claim 1, wherein, The CPU is further configured to receive the accuracy of the feedback comparison result, correct the correspondence between the pre-set fault tolerance weight and the sampling ratio based on the accuracy, and increase the plurality of pre-set sampling ratios corresponding to the fault tolerance weight when the accuracy is lower than a pre-set threshold. correct the pre-created data statistical model based on the accuracy, and increase the probability of selecting the plurality of matching algorithms when the accuracy is higher than a pre-set threshold.

8. The system of claim 1, wherein, The CPU is further configured to perform parallel or serial calculation on the first data and the second data by using the plurality of matching algorithms, compare the calculation results, and send the first data and the second data to the hash chip if the calculation results are consistent. receive and compare the hash calculation result; and determine whether the first data and the second data are identical based on the comparison result.

9. A data comparison method, characterized by, A system for data comparison, the system comprising: a hash chip and a central processing unit (CPU); the hash chip and the CPU are electrically connected; the method comprising: The CPU obtains first data and second data to be compared; identifies a target algorithm and a target sampling ratio corresponding to the first data and the second data through a pre-created data statistical model; wherein the target algorithm includes a hash algorithm and a plurality of matching algorithms in a plurality of pre-set algorithms; the data statistical model is analyzed and automatically optimized based on historical data comparison records, and the historical data comparison records include one or more factors such as difference detection capability, calculation delay, calculation amount, calculation complexity and calculation power consumption of various algorithms; the data statistical model is used to dynamically screen a plurality of target algorithms according to the factors, and determine the execution order of the plurality of target algorithms when calculated in parallel or in series; and is also used to adjust the target sampling ratio according to the dynamic relationship between historical difference detection capability and calculation delay, automatically increase the target sampling ratio if the difference detection capability is lower than the expected value, automatically reduce the target sampling ratio if the difference detection capability is higher than the expected value and the calculation delay is greater than the pre-set delay, or determine the target sampling ratio according to the corresponding relationship between the pre-set fault tolerance weight of data and the sampling ratio; The CPU samples the first data according to the target sampling ratio to obtain third sampling data, and samples the second data according to the target sampling ratio to obtain fourth sampling data, calculates the third sampling data and the fourth sampling data through the plurality of matching algorithms in parallel or in series, and sends the third sampling data and the fourth sampling data to the hash chip; The hash chip receives the third sampling data and the fourth sampling data; performs hash calculation on the third sampling data and the fourth sampling data; and sends the calculated hash calculation result to the CPU; The CPU receives the hash calculation result of the third sampling data and the hash calculation result of the fourth sampling data, and the parallel or serial calculation result, and compares the results; and judges whether the first data and the second data are the same according to the comparison result.

Images