Data synchronization and comparison method and device and electronic equipment
Through the combination of data chunking and hash comparison, the existing data comparison methods are solved, and efficient and accurate data consistency maintenance is achieved, and it is suitable for a variety of database systems.
Patent Information
- Application Number
- CN202510347582.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-07-01
AI Technical Summary
The existing data comparison methods are inefficient, have poor accuracy and strong dependence in large data volume scenarios, and cannot effectively deal with the problems of complex data structures and inaccurate timestamps.
Using a combination of data chunking and hash comparison, data chunking and hash calculations are provided by combining preset rules, combining item-by-item and field-by-field comparisons, manual and automatic repair mechanisms are provided, suitable for multiple database systems, and parallel processing is utilizing asynchronous multithreading.
It improves the efficiency and accuracy of data comparison, reduces computing resource consumption, is suitable for various complex data structures, reduces storage costs, supports multiple database systems, and achieves efficient maintenance of data consistency.
Smart Images

Figure CN120234372A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of data synchronization, and particularly relates to a method, device, computer-readable storage medium, and electronic device for data comparison in data synchronization. Background Art
[0002] In today's information society, data comparison after data synchronization is an important link to ensure data consistency and integrity. The commonly used data comparison methods are as follows: (1) Full-scale comparison method: Compare all the data at the source end with the data at the target end. This method is suitable for scenarios with a small amount of data. When the data volume is huge, full-scale comparison will consume a large amount of time and computing resources, resulting in low efficiency.
[0003] (2) Incremental comparison method: Only compare the part of the data that has changed, thereby reducing the amount of data to be compared. However, it is necessary to accurately identify data changes, and it is prone to errors when dealing with complex data structures.
[0004] (3) Timestamp comparison method: Determine whether the data has changed by recording the timestamps of the data. However, it depends on the accuracy of the timestamps and cannot handle the situation where the timestamps are the same but the content is different.
[0005] In summary, the existing data comparison methods still have many defects and deficiencies, including: (1) Low efficiency: The full-scale comparison method is inefficient when dealing with a large amount of data.
[0006] (2) Poor accuracy: The incremental comparison method and the timestamp comparison method are prone to errors when dealing with complex data structures.
[0007] (3) Strong dependence: The timestamp comparison method depends on the accuracy of the timestamps and has certain limitations.
[0008] Therefore, developing a more efficient, accurate, and applicable data comparison method has become an urgent need in this industry. Summary of the Invention
[0009] To address the above problems, this application proposes a new method and device for data comparison in data synchronization, aiming to solve the problems of low efficiency, poor accuracy, and strong dependence of the existing data comparison methods.
[0010] This application mainly adopts the following technical strategies: (1) Data chunking and dynamic adjustment mechanism Adopt the data chunking strategy, divide the source-end data and the target-end data into multiple data chunks according to preset rules (such as primary key range, hash value, time interval, etc.), and introduce a mechanism for dynamically adjusting the chunk size.
[0011] Advantages: By reducing the amount of data for comparison in chunks, the comparison efficiency is improved; The chunk size is dynamically adjusted according to the data volume and server resources to optimize resource utilization, reduce storage space occupation and calculation amount, and lower storage costs; Asynchronous multi-threaded parallel chunking and comparison are adopted to significantly improve the overall efficiency.
[0012] (2) Combining hash comparison and record comparison A two-stage comparison method (dual comparison mechanism) combining hash comparison and record comparison is proposed.
[0013] Hash comparison: Calculate the hash value for each data chunk (such as the MD5 algorithm), and quickly determine the inconsistent data chunks by comparing the hash values.
[0014] Record comparison: Compare each inconsistent data chunk item by item and field by field to accurately locate the inconsistent records and fields.
[0015] Advantages: Hash comparison quickly filters out inconsistent data chunks and narrows the comparison scope; Item-by-item comparison and field-by-field comparison ensure the accuracy of the comparison and avoid misjudgment; It does not rely on external factors such as timestamps and is applicable to various complex data structures.
[0016] (3) Hash calculation and comparison supporting multiple databases For different databases (such as MySQL, Oracle, etc.), unified hash calculation and comparison methods are provided to support cross-database data synchronization comparison.
[0017] Advantages: Applicable to multiple database systems with strong scalability; Hash calculation is implemented through SQL statements and built-in functions, and the operation is simple.
[0018] (4) Flexible mechanism for data repair Two data repair methods, manual repair and automatic repair, are provided.
[0019] Manual repair: Support users to update or delete inconsistent data through custom SQL statements.
[0020] Automatic repair: Automatically delete inconsistent data at the target end and re-insert the data from the source end.
[0021] Advantages: Can meet the repair requirements of different scenarios; Automatic repair reduces manual intervention and improves the repair efficiency.
[0022] (5) Asynchronous multi-threaded parallel processing Asynchronous multi-threaded technology is adopted to parallel process data chunks and data comparison to avoid the performance bottleneck of a single thread.
[0023] Advantages: Significantly improves the efficiency of data chunking and comparison; Makes full use of server resources and improves the overall performance.
[0024] Generally speaking, this application provides a data synchronization and data comparison method, including the following steps: (1) Data chunking The starting point of data synchronization is to process the source data and the target data in chunks according to preset rules. This process designs the chunking strategy based on the structural characteristics of the data and business requirements. For example, data chunks can be divided according to the unique identifier of the data record (such as the primary key) to ensure that each data chunk contains several data records with consecutive identifiers, thus maintaining the logical integrity of the data. At the same time, the size of the chunks also needs to be flexibly adjusted according to the actual application scenario to achieve the best comparison efficiency. For example, when processing large-scale data, the chunk size can be appropriately increased to reduce the number of chunks and the computational overhead during the comparison; while for data synchronization scenarios with high real-time requirements, smaller chunks can be used to more quickly detect and process data differences.
[0025] (2) Hash calculation After the data chunking is completed, the next key step is to perform hash calculation on each data chunk. The hash algorithm is an algorithm that maps an input of any length of data to an output of a fixed length, and has the characteristics of fast calculation and difficulty in reverse. The hash algorithm selected in this solution can ensure that when the content of the data chunk changes, its corresponding hash value will also change accordingly, thus providing a reliable basis for subsequent comparison. Specifically, for each data chunk at the source and target ends, calculate its hash value respectively and store these hash values for subsequent comparison operations. Through hash calculation, the complex data chunk comparison problem can be transformed into a simple hash value comparison problem, greatly improving the comparison efficiency and reducing the consumption of storage and computing resources.
[0026] (3) Hash comparison Hash comparison is one of the core links of this solution. In this stage, the hash values of the source data chunks and the target data chunks are compared one by one. If the hash values of a certain data chunk at the source and target ends are the same, it can be initially judged that the data chunk is consistent at both ends and no further comparison is required; while if the hash values are different, it indicates that there are differences in this data chunk at both ends and it needs to enter the next detailed comparison. This preliminary screening mechanism based on hash values can quickly locate the data chunks that may have inconsistent data, avoiding a comprehensive comparison of all data chunks record by record and field by field, and significantly improving the efficiency and pertinence of the comparison. In addition, in order to further improve the accuracy of the comparison, some optimization strategies can be introduced during the hash comparison process, such as using a multi-level hash algorithm to perform a more detailed hash value comparison on the initially screened inconsistent data chunks to reduce the possibility of misjudgment.
[0027] (4) Record comparison For data blocks determined to be inconsistent through hash comparison, more detailed record comparison is required. The purpose of record comparison is to precisely identify which specific records in the data block are different and the specific content of the differences. This process includes two levels of comparison: record-by-record comparison and field-by-field comparison. In the record-by-record comparison stage, each record in the source data block is compared with the record at the corresponding position in the target data block to determine whether there are overall differences in the records, such as the absence, duplication, or inconsistent order of records. For records with differences, further field-by-field comparison is carried out. Field-by-field comparison is to compare each field value in the record one by one to determine which specific field values are inconsistent between the source and the target, so as to accurately locate the source of the data difference. Through this detailed comparison of records and fields one by one, the detailed situation of inconsistent records in the data block can be clearly presented, providing an accurate basis for subsequent data repair.
[0028] (V) Data Repair Data repair is the ultimate goal of the data synchronization data comparison method and a key link to ensure data consistency. According to the specific situation of the inconsistent records determined by the previous record comparison, this application provides flexible data repair means. The data repair module can provide two ways, manual repair and automatic repair, to correct the target data according to the comparison result. Manual repair means that the user directly modifies the target data according to the comparison result. This method is applicable to some complex data difference situations, or when there may be risks in automatic repair. The user can accurately repair the data based on their own business knowledge and experience. Automatic repair is to automatically correct the target data by the system according to the preset repair rules. For example, for simple field value differences, the system can automatically update the corresponding field value in the target according to the value in the source data; for the problem of missing or duplicate records, the system can automatically add or delete the corresponding records. The automatic repair method can greatly improve the efficiency of data repair and reduce manual intervention, especially suitable for large-scale data synchronization scenarios. By combining the two methods of manual repair and automatic repair, various data difference situations can be flexibly handled, thus ensuring the final consistency of the data and ensuring that the data always remains accurate, complete, and synchronized between the source and the target.
[0029] To implement the above data synchronization data comparison method, this application also provides a data synchronization data comparison device, which consists of multiple functional modules. Each module works together to complete the whole process of data synchronization data comparison.
[0030] (I) Data Chunking Module The data chunking module is one of the basic modules of the device. Its main function is to chunk the source - side data and the target - side data according to preset rules. This module can flexibly design chunking strategies based on the structural characteristics and business requirements of the data, and divide the data into data chunks of appropriate size and complete logic. It supports multiple chunking methods, such as chunking based on the primary key of data records, chunking based on timestamps, etc., to meet the data synchronization requirements in different application scenarios.
[0031] (2) Hash comparison module The hash comparison module is one of the core modules of the device, undertaking the important task of calculating and comparing the hashes of data chunks. This module integrates an efficient hash algorithm, which can quickly calculate the hash value of each data chunk and accurately generate the corresponding hash value. During the hash comparison process, this module can compare the hash values of data chunks on the source side and the target side one by one, quickly locate the data chunks with differences, and feedback the comparison results to the subsequent record comparison module. To improve the accuracy and efficiency of the comparison, the hash comparison module also has some optimization functions, such as supporting multi - level hash algorithms, automatically adjusting hash calculation parameters, etc., to adapt to different data characteristics and comparison requirements. In addition, this module can monitor and handle abnormal situations during the hash comparison process, such as hash value calculation errors, data chunk loss, etc., to ensure the stability and reliability of the comparison process.
[0032] (3) Record comparison module The record comparison module is a key module in the device for processing inconsistent data chunks. When the hash comparison module determines that a certain data chunk has differences, the record comparison module will conduct a detailed line - by - line and field - by - field comparison of the records in this data chunk. This module can accurately find out which specific records in the data chunk have differences and the specific content of the differences, including record omission, duplication, field value inconsistency, etc. The record comparison module adopts an efficient comparison algorithm, which can complete the comparison operation of a large number of records in a short time and present the comparison results in a clear and intuitive way, providing an accurate basis for the data repair module. In addition, this module also has a certain degree of fault tolerance and flexibility, and can handle some special situations, such as inconsistent data formats, missing fields, etc., to ensure the accuracy and integrity of the comparison results.
[0033] (4) Data repair module The data repair module is the final execution module of the device. Its purpose is to repair the target - side data according to the comparison result of the record comparison module to ensure the final consistency of the data. This module provides two repair methods: manual repair and automatic repair. Users can choose the appropriate repair method according to the actual situation. The manual repair method allows users to directly modify the target - side data. Users can intuitively view the comparison result through the operation interface provided by the device and perform precise repair operations on the data based on their business knowledge and experience. The automatic repair method corrects the target - side data automatically by the system according to the preset repair rules. The data repair module can intelligently identify different data difference situations and automatically execute repair operations according to the corresponding repair rules, such as updating field values, adding or deleting records, etc. This module also has data backup and recovery functions. Before performing the automatic repair operation, it will automatically back up the target - side data to prevent data loss caused by unexpected situations during the repair process. At the same time, after the repair is completed, it can also verify the repair result to ensure the correctness of the repair operation and the consistency of the data.
[0034] Specifically, to achieve the above - mentioned purpose, the present application provides the following technical solutions: The first aspect of the present application provides a method for data synchronization and data comparison, and the method includes: Data chunking: Chunk the source - side data and the target - side data according to preset rules. Each data chunk contains several data records; Hash calculation: Calculate the hash value for each data chunk to generate the corresponding hash value; Hash comparison: Compare the hash values of the source - side data chunk and the target - side data chunk to determine the inconsistent data chunks; Record comparison: Compare each record and each field in the inconsistent data chunks one by one to determine the specific inconsistent records; Data repair: Repair the target - side data according to the record comparison result to ensure the final consistency of the data.
[0035] Optionally, in the method of the present application, in the data chunking step, the preset rules include at least one of the following: (1) Chunk by primary key range; (2) Chunk by hash value; (3) Chunk by time.
[0036] Optionally, in the method of the present application, the data chunking step further includes: Adopt the method of dynamically adjusting the chunk size, and adjust the chunk size in real - time according to the data volume and server resource situation to balance the processing speed and the consumption of server resources; The division of data chunks is performed in parallel in an asynchronous multi-threaded manner. When a data chunk is determined, subsequent data comparison operations are immediately carried out to improve the comparison efficiency.
[0037] Optionally, in the method of the present application, in the hash calculation step, the MD5 algorithm is used to calculate the hash value of each data block.
[0038] Optionally, in the method of the present application, the hash comparison step further includes: performing further hash value comparison on the initially screened inconsistent data blocks using a multi-level hash algorithm to reduce the possibility of misjudgment.
[0039] Optionally, in the method of the present application, in the record comparison step, the comparison item by item includes the following steps: Concatenate the fields of a single record into a string; Use a hash algorithm to calculate the hash value of the concatenated string; Compare the hash values of the records at the source end and the target end to determine whether the records are consistent; In the record comparison step, the comparison field by field includes: for the data records with inconsistent hash values found in the comparison item by item, compare the string values of each field in sequence according to the field order to determine the inconsistent fields.
[0040] Optionally, in the method of the present application, in the data repair step, two methods of manual repair and automatic repair are provided. Among them, manual repair supports the user to update or delete the inconsistent data at the target end through custom SQL statements, and automatic repair automatically corrects the data at the target end by the system according to preset rules.
[0041] The second aspect of the present application provides a device for data synchronization and data comparison, and the device includes: A data chunk module: used to perform chunk processing on the source end data and the target end data according to preset rules, so that each data chunk contains several data records; A hash comparison module: used to perform hash calculation on each data chunk to generate corresponding hash values, and then compare the hash values of the source end data chunk and the target end data chunk to determine the inconsistent data chunks; A record comparison module: used to perform item-by-item comparison and field-by-field comparison on the records in the inconsistent data chunks to determine the specific inconsistent records; A data repair module: used to repair the target end data according to the record comparison result to ensure the final consistency of the data.
[0042] When the device runs, it implements the steps of the aforementioned data synchronization and data comparison method.
[0043] The third aspect of the present application provides an electronic device, including: a memory and a processor; Memory: for storing computer programs; Processor: for executing the computer program to implement the steps of the data synchronization and data comparison method described above.
[0044] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization and data comparison method described above are implemented.
[0045] In summary, the data synchronization and data comparison method proposed in the present application significantly improves the efficiency, accuracy, and applicability of data synchronization and data comparison through efficient data chunking and hashing calculation strategies, a dual comparison mechanism, and a design concept independent of external factors, providing reliable technical support for data consistency maintenance. This method has the following advantages: (1) High efficiency Through the optimized strategy of data chunking and hashing calculation, this solution effectively reduces the amount of data in the data comparison process, reduces the computational complexity of the comparison, improves the comparison efficiency, and is particularly suitable for large-scale data synchronization scenarios.
[0046] (2) Accuracy This solution adopts a dual comparison mechanism combining hash comparison and record comparison, improves the accuracy of the comparison result, reduces the error rate, and ensures that data inconsistencies can be accurately identified and repaired during the data synchronization process.
[0047] (3) Independence The data synchronization and data comparison method of the present application does not depend on external factors such as timestamps and data update orders, but performs comparisons based on the data content itself. It is applicable to various complex data structures and data storage methods, has wide applicability, and can meet the data synchronization requirements in different systems and different application scenarios. Especially in scenarios with complex data structures, frequent data updates, and difficulty in relying on external markers such as timestamps, it demonstrates strong advantages.
[0048] Other features and advantages of the present application will be elaborated in detail in the subsequent specification, or can be understood by implementing the related technical solutions of the present application. The objectives and other advantages of the present application can be achieved through the technical features and technical means clearly pointed out in the specification, claims, and drawings, and obtained through the implementation process of these technical contents. Description of the Drawings
[0049] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the accompanying drawings only show some embodiments of the present application. For those skilled in the art, without creative work, other relevant drawings can be deduced based on these drawings.
[0050] Figure 1 This is the overall design architecture diagram of the solution of the present application.
[0051] Figure 2 This is the overall implementation flowchart of the method for data synchronization and data comparison of the present application.
[0052] Figure 3 This is the composition structure diagram of the device for data synchronization and data comparison of the present application.
[0053] Figure 4 This is the schematic structural diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners
[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. It should be clear that the described embodiments are only some embodiments of the present application, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative work fall within the protection scope of the present application.
[0055] In this document, the term "including" and any form of its deformation (such as "including", "including") are open-ended expressions and should be understood as "including but not limited to", that is, the listed content is not an exhaustive list and may also include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on", that is, the referred basis or condition may not be the only factor and may also involve other relevant factors. The term "an embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation and there may be other similar embodiments.
[0056] In the present application, when the terms "a" and "multiple" are used to modify related elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "a" should be understood as "at least one", and "multiple" should be understood as "at least two". Those skilled in the art should reasonably interpret these terms according to the semantic and logical relationships in the context to ensure that they cover the possibility of "one or more".
[0057] Figure 1 and Figure 2The overall implementation process of the data synchronization and comparison method provided by this application is shown below, including the following steps: S1. Data chunking: The source-side data and the target-side data are chunked according to a preset rule, and each data chunk contains several data records; S2. Hash calculation: Hash calculation is performed on each data chunk to generate the corresponding hash value; S3. Hash comparison: The hash values of the source-side data chunks and the target-side data chunks are compared to determine the inconsistent data chunks; S4. Record comparison: The records in the inconsistent data chunks are compared one by one and field by field to determine the specific inconsistent records; S5. Data repair: According to the record comparison result, the target-side data is repaired to ensure the final consistency of the data.
[0058] To more clearly illustrate the technical solution of this application, the following will be further described through examples in specific scenarios.
[0059] The device for data synchronization and comparison provided by this application includes the following functional modules: (I) Data chunking module The function of the data chunking module is to split the source-side data and the target-side data into smaller data chunks according to a preset rule, so as to process and compare the data more efficiently.
[0060] Common chunking strategies include: (1) Range-based chunking: The data is divided into multiple chunks according to a certain numerical range (such as time, ID, etc.).
[0061] (2) Hash-based chunking: The data is evenly distributed into multiple chunks through a hash function to ensure the balance of data distribution.
[0062] (3) Key-value-based chunking: Chunking is performed according to a certain key value (such as user ID) to ensure the logical consistency of the data chunks.
[0063] In actual applications, the tables and table structures used are all different. In order to be able to compare the data of various table structures, in this solution, the chunking rule adopted is: (1) Chunking by primary key range: If the table has a primary key (such as an auto-incrementing ID, etc.), chunking can be performed according to the range of the primary key. For example, the records with ID from 1 to 10000 are used as the first chunk, and the records from 10001 to 20000 are used as the second chunk, and so on.
[0064] (2) Chunking by hash value: If there is no primary key, a hash function can be used to calculate the hash value of a certain field (such as timestamp, UUID, etc.), and then chunking is performed according to the range of the hash value.
[0065] (3) Blocking by time: If there is a time field in the table (such as creation time, etc.), it can be blocked by time intervals. For example, the data of each day or each hour can be used as a block.
[0066] By adopting the above blocking rules, this solution can be applied to most table structures.
[0067] In addition, in order to balance the processing speed and the consumption of server resources, a dynamic adjustment of the block size is adopted in this solution. Based on the data volume and server resources, the block size is dynamically adjusted according to the following principles: (1) When the data volume is very large, the block size will be appropriately increased, which is beneficial to reducing the block comparison time and improving the comparison efficiency.
[0068] (2) When the server resources are tense, the block size will also be increased accordingly, reducing the number of blocks, which is beneficial to reducing the storage space occupation, reducing the storage cost, and reducing the calculation amount.
[0069] In addition, when the data volume in the table is very large, there will be the following performance problems when using a single thread for data blocking: The data blocks that have been divided in the table need to wait until all the data blocks in the table are completely divided before starting the data comparison, resulting in low resource utilization and greatly reducing the data comparison efficiency. In response to the above performance problems of the single thread, in this solution, an asynchronous multi-threaded method is adopted to parallelize the division of data blocks. When a data block is determined, the subsequent data comparison operation is immediately carried out, without waiting until all the data blocks are divided before starting the comparison, thus significantly improving the overall comparison efficiency.
[0070] (2) Hash comparison module The hash comparison module mainly takes data blocks as units and compares the data blocks according to the hash values to determine the inconsistent data blocks.
[0071] Hash comparison is to generate the unique identifier hash value of the data block through the hash algorithm, and then judge whether the data blocks are consistent by comparing the hash values.
[0072] The hash comparison process includes two links: hash calculation and hash comparison.
[0073] (1) Hash calculation When performing hash calculation, the data in a data block is calculated as a whole through the hash algorithm to generate the corresponding hash value.
[0074] In this embodiment, the MD5 algorithm is adopted to calculate the hash value.
[0075] The MD5 algorithm is a widely used hashing algorithm that can be used in data verification scenarios, especially suitable for data consistency checks and file integrity verification. Its application scenarios include: 1) Data consistency check: During database migration, backup recovery, or data transmission, the MD5 algorithm can be used to verify data consistency. By comparing the MD5 values of the source data and the target data, it can be confirmed whether the data is complete and error-free.
[0076] 2) File integrity verification: Files may be damaged during transmission or storage due to network anomalies or disk failures. By calculating the MD5 value of the file, it can quickly detect whether the file is complete and avoid using damaged data.
[0077] Its advantages are: 1) Fast calculation speed: The MD5 algorithm is efficient and suitable for quickly calculating the hash value of a file, especially suitable for verifying large files.
[0078] 2) Simple implementation: The MD5 algorithm is easy to implement, and most databases and systems have built-in MD5 functions or calculation tools.
[0079] 3) Wide application: MD5 is widely used in fields such as file verification and software distribution and is a mature technology.
[0080] This embodiment takes the source Mysql database and the target Oracle database as examples to illustrate the process of hash calculation: Assume that the data is divided into blocks according to the primary key range. The data range of one data block is 1 - 10000. Write hash calculation scripts for the source and target ends respectively: Table 1 Example table of source and target table information and field mapping relationship
[0081] Mysql side: Use SQL statements to query data according to the block rule and calculate the hash value using the MD5 algorithm. For example, in Mysql, the GROUP_CONCAT function can be used to concatenate multiple fields in the data set.
[0082]
[0083] Among them, id, name, and age are fields in the source Mysql table.
[0084] Oracle side: Similarly, use SQL statements to query data according to the block rule and calculate the hash value using the MD5 algorithm. Among them, in Oracle, the string concatenation operator '||' can be used to concatenate multiple fields in the data set.
[0085]
[0086] Among them, ID, NAME, and AGE are fields of the target Oracle table.
[0087] (2)Hash comparison Hash comparison is to compare the hash values of the source data and the target data to determine inconsistent data blocks. If the hash values of two data blocks are the same, these two data blocks are considered consistent; if the hash values are different, these two data blocks are considered inconsistent.
[0088] In this solution, when two data blocks with differences are found through program comparison, the data blocks with differences at the source and the target will be marked and then handed over to the record comparison module for further processing.
[0089] (III)Record comparison module The function of the record comparison module is to perform a detailed comparison of the data blocks with differences to find out the records with differences.
[0090] In order to find out the records with differences in two data blocks efficiently and accurately, in this solution, the methods of comparing one by one and comparing field by field are adopted in sequence.
[0091] (1)Comparison one by one In the link of comparison one by one, for the data set in a data block, each row of records is compared one by one with a single record as the unit to determine inconsistent records.
[0092] The implementation process is as follows: 1) Concatenate the fields of the single record to be compared into a string. For example, in Mysql, the CONCAT_WS function can be used to concatenate multiple fields of a record.
[0093]
[0094] 2) Use the built-in MD5 function of the database to calculate the MD5 hash value of the concatenated string.
[0095] Among them, the MD5 functions of different databases are as follows:
[0096] After comparison, if the hash values of two records are the same, these two records are considered consistent; if the hash values are different, these two records are considered inconsistent and need to continue with the comparison field by field.
[0097] (2)Comparison field by field When records with inconsistent hash values are found after comparison one by one, the comparison field by field link is entered.
[0098] The per-field comparison process compares each record field by field in the order of the fields to determine the inconsistent fields.
[0099] During the comparison process, only the string values of the fields need to be compared in the order of the fields. Since the per-field comparison requires comparing each field one by one, the efficiency is relatively low and it is suitable for scenarios with a small amount of data. Generally, after data synchronization, there are few different datasets, so the amount of data entering the per-field comparison process is also small, and the overall impact on performance is not significant.
[0100] (4) Data repair module Manual repair: For a small amount of inconsistent data, it can be repaired manually, and users are supported to update or delete inconsistent data at the target end through custom SQL statements.
[0101] Automatic repair: When using automatic repair, the program will automatically delete the inconsistent data at the target end and then re-insert it into the target end based on the source data.
[0102] Figure 3 The figure shows a device for data synchronization and comparison according to the present application, including: Data chunking module: used to chunk the source data and the target data according to preset rules so that each data chunk contains several data records; Hash comparison module: used to calculate the hash value for each data chunk, generate the corresponding hash value, and then compare the hash values of the source data chunk and the target data chunk to determine the inconsistent data chunks; Record comparison module: used to compare each record in the inconsistent data chunks one by one and field by field to determine the specific inconsistent records; Data repair module: used to repair the target data according to the record comparison result to ensure the final consistency of the data.
[0103] When the above device runs, it implements the steps of the data synchronization and comparison method disclosed in the present application.
[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementations of devices, methods, and computer program products according to various embodiments of the present application, including system architectures, functions, and operations. In these figures, each block may represent a module, a program segment, or a part of the code, which contains one or more executable instructions for implementing the specified logical function. It should be noted that each block in the block diagram and / or flowchart, as well as combinations of these blocks, can be implemented using a dedicated hardware-based system to perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.
[0105] AsFigure 4 As shown in Figure 4 , an embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing computer programs executable by the processor, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete communication with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above data synchronization and data comparison method.
[0106] It can be understood that in addition to including a memory and a processor, the electronic device may further include an input device (such as a keyboard), an output device (such as a display), and other communication modules. These input devices, output devices, and other communication modules communicate with the processor through an I / O interface (i.e., an input / output interface).
[0107] The operations of the present application can be implemented by writing computer program code using one or more programming languages or combinations thereof. The programming languages include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; Conventional procedural programming languages, such as the "C" language or similar programming languages.
[0108] The execution modes of the program code include but are not limited to: Fully executed on the user's computer; Partially executed on the user's computer and partially executed on a remote computer; Executed as an independent software package; Fully executed on a remote computer or server.
[0109] In scenarios involving remote computers, the remote computer can be connected to the user's computer through any type of network, and the network includes but is not limited to a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, such as using the Internet for connection.
[0110] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute each step of the data synchronization and data comparison method disclosed in the present application.
[0111] In the context of the present application, a computer-readable storage medium refers to a tangible medium that can store computer program code and related data. Specific examples include but are not limited to the following: (1) Portable computer disks: Removable magnetic storage media such as floppy disks.
[0112] (2) Hard disks: Fixed storage devices including mechanical hard disks and solid-state drives.
[0113] (3) Random Access Memory (RAM): Volatile storage media for temporarily storing data and program code.
[0114] (4) Read-Only Memory (ROM): Non-volatile storage media for storing fixed programs and data.
[0115] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: Non-volatile storage media that support multiple erasures and programming.
[0116] (6) Fiber optic storage devices: Storage media based on fiber optic technology.
[0117] (7) Portable Compact Disc Read-Only Memory (CD-ROM): Read-only media for storing data in the form of optical discs.
[0118] (8) Optical storage devices: Storage media based on optical principles such as DVDs and Blu-ray discs.
[0119] (9) Magnetic storage devices: Storage media based on magnetic principles such as magnetic tapes and disks.
[0120] (10) Any suitable combination of the above: For example, combining multiple storage media to meet different storage requirements.
[0121] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.
[0122] In particular, according to the embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, the embodiments of this application relate to a computer program product that includes a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for performing the method of data synchronization and data comparison disclosed in this application. When the computer program is executed by a processing device, it can implement the above functions defined in the embodiments of this application.
[0123] Although the above description contains several specific implementation details, these details should not be construed as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, the present application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept.
[0124] Those skilled in the art should also understand that they can modify the technical solutions described in the foregoing embodiments or perform equivalent replacements on some of the technical features without departing from the spirit and scope of the technical solutions of the embodiments of the present application. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for data synchronization and data comparison, characterized in that: The method comprises: Data block: The source data and the target data are divided into blocks according to preset rules. Each data block contains several data records. Hash calculation: Perform hash calculation on each data block to generate the corresponding hash value; Hash comparison: compare the hash values of the source data block and the target data block to identify inconsistent data blocks; Record comparison: compare the records in the inconsistent data blocks one by one and field by field to identify the specific inconsistent records; Data repair: Repair the target data based on the record comparison results to ensure the final consistency of the data.
2. The method according to claim 1, characterized in that In the data segmentation step, the preset rules include at least one of the following: (1) Block by primary key range; (2) Block by hash value; (3) Divide into blocks by time.
3. The method according to claim 1, characterized in that The data segmentation step also includes: Dynamically adjust the block size in real time according to the data volume and server resources to balance the processing speed and server resource consumption; The data blocks are divided in parallel using asynchronous multi-threading. When a data block is determined, the subsequent data comparison operation is immediately performed to improve the comparison efficiency.
4. The method according to claim 1, characterized in that: In the hash calculation step, the MD5 algorithm is used to calculate the hash value of each data block.
5. The method according to claim 1, characterized in that The hash comparison step also includes: using a multi-level hash algorithm to perform further hash value comparison on the inconsistent data blocks initially screened out to reduce the possibility of misjudgment.
6. The method according to claim 1, characterized in that In the record comparison step, the record comparison includes the following steps: Concatenate the fields of a single record into a string; Use the hash algorithm to calculate the hash value of the concatenated string; Compare the hash values of the source and target records to determine whether the records are consistent; In the record comparison step, the field-by-field comparison includes: for each data record with inconsistent hash values found in the comparison, the string value of each field is compared in sequence according to the field order to determine the inconsistent fields.
7. The method according to claim 1, characterized in that In the data repair step, two methods, manual repair and automatic repair, are provided. Manual repair supports users to update or delete inconsistent data on the target side through custom SQL statements, while automatic repair allows the system to automatically correct the target side data according to preset rules.
8. A device for data synchronization and data comparison, characterized in that: include: Data block module: used to block the source data and the target data according to preset rules, so that each data block contains several data records; Hash comparison module: used to perform hash calculation on each data block, generate the corresponding hash value, and then compare the hash values of the source data block and the target data block to determine the inconsistent data blocks; Record comparison module: used to compare records in inconsistent data blocks line by line and field by field to identify specific inconsistent records; Data repair module: used to repair the target end data according to the record comparison results to ensure the final consistency of the data.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for data synchronization and data comparison as described in any one of claims 1 to 7 are implemented.
10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the method for data synchronization and data comparison as described in any one of claims 1-7.
Citation Information
Cited By
Data synchronization method and product
CN121051180A
