Restart mass data comparison method, device and system and medium

Abstract

The invention provides a restart mass data comparison method, device and system and a medium. The method comprises the steps that data tables needing to be compared are obtained based on a source end database and a target end database; splitting the data table into a plurality of data segments, and generating corresponding query statements; executing the query statement to compare the data segments, and generating difference data; performing secondary hash processing on the query statements of the data segments subjected to data comparison to obtain hash values, and storing the hash values in a secondary hash set; when a data comparison restart task is identified, reading a secondary hash set of the data comparison task, and obtaining data segments needing to be compared again; and executing the query statements of the data segments needing to be compared again to obtain difference data. According to the invention, by adopting a sectional comparison method, the efficiency of data comparison is improved, and when comparison needs to be carried out again in the data comparison process, the corresponding hash value can be read to locate the data section needing to be compared again, so that the convenience of data comparison is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device, system and medium for restarting mass data comparison. Background Art

[0002] In the current era of data explosion, the country has clearly proposed to accelerate digital development and build a digital China, and the importance of data has become increasingly prominent. However, the preservation and processing of massive data is still a major challenge facing the industry, and data backup is an important means to ensure data reliability.

[0003] A large amount of data maintenance involves backup. Differences are inevitable when backing up from one medium to another. In order to ensure the reliability of the backup data, data comparison poses a challenge. Comparing massive data is a time-consuming and complicated process, especially when encountering emergencies such as power outages, network disconnections, and server crashes. The process may be interrupted. Most current data comparison software often chooses to restart the comparison when dealing with these problems, which not only wastes previous time, but may also encounter the same problem again when re-comparing. This situation seriously affects the user experience and emotions, especially when time is limited, it is difficult to ensure that the data comparison work is completed in a timely manner. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for restarting massive data comparison. The present invention improves the efficiency of data comparison by adopting a segmented comparison method. When the data comparison process needs to be re-compared, the data segment that needs to be re-compared can be quickly located by reading the corresponding hash value, thereby enhancing the convenience of data comparison.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A first aspect of the present invention provides a method for restarting mass data comparison, comprising the following steps:

[0007] Obtain the data tables that need to be compared based on the source database and the target database;

[0008] Splitting the data table into a plurality of data segments, and generating a query statement for each data segment;

[0009] Execute the query statement of the data segment to compare the data segment and generate difference data;

[0010] Performing secondary hash processing on the query statement of the data segment that has completed the data comparison, obtaining a hash value and storing the hash value in a secondary hash set;

[0011] When a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data.

[0012] In one embodiment, when a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data, including:

[0013] Read the secondary hash set of the data comparison task, and determine whether the secondary hash set is an empty set;

[0014] When the secondary hash set is an empty set, the data comparison task is started for the first time;

[0015] When the secondary hash set is not an empty set, the data comparison task is restarted.

[0016] In one embodiment, splitting the data table into a plurality of data segments and generating a query statement for each data segment includes:

[0017] Creating a histogram in a source database or a target database according to the data table;

[0018] Splitting the data table into a specified number of data segments;

[0019] Generate query statements for the source database and the target database for each data segment based on the statistical information of the histogram.

[0020] In one embodiment, performing secondary hash processing on the query statement of the data segment that has completed the data comparison to obtain a hash value and storing the hash value in a secondary hash set includes:

[0021] Perform the first hash processing on the source database query statement of each data segment to obtain the first hash value;

[0022] Performing a second hash process on the first hash value to shorten the length of the first hash value and obtain a second hash value;

[0023] The second hash value is stored in a secondary hash set.

[0024] In one embodiment, when a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data, including:

[0025] When the secondary hash set is not an empty set, the data comparison task is restarted, and the data comparison restart task is identified;

[0026] Read the secondary hash set of the data comparison task, and determine whether the second hash value exists in the secondary hash set;

[0027] When the second hash value exists in the secondary hash set, the data segment corresponding to the second hash value has completed the data comparison task;

[0028] When the second hash value does not exist in the secondary hash set, the data segment corresponding to the second hash value has not completed the data comparison task, and the data segment that needs to be compared again is obtained;

[0029] Execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

[0030] In one embodiment, executing the query statement of the data segment to be re-compared to obtain the difference data specifically includes:

[0031] Execute query statements on the data segments of unfinished comparison tasks to perform data comparison;

[0032] The difference data generated by the data comparison is sent to the blocking queue for storage until the data comparison task of the data segment is completed.

[0033] In one embodiment, the step of sending the difference data generated by the data comparison to a blocking queue for storage until the data comparison task of the data segment is completed further includes:

[0034] After the data comparison task is completed, the second hash value corresponding to the data segment is stored in the secondary hash set;

[0035] Determine whether the data comparison task of the data table is completed according to the second hash value in the second hash set;

[0036] When the data comparison task of the data table is completed, the difference data is extracted from the blocking queue, the difference data is summarized and processed, and a data report is generated from the summarized difference data and transmitted to a designated location for storage.

[0037] A device for restarting mass data comparison, the device comprising:

[0038] The data table module is used to obtain the data table that needs to be compared based on the source database and the target database;

[0039] A data splitting module, used to split the data table into several data segments and generate a query statement for each data segment;

[0040] A data comparison module, used for executing the query statement of the data segment to compare the data segment and generate difference data;

[0041] A hash processing module, used for performing secondary hash processing on the query statement of the data segment that has completed the data comparison, obtaining a hash value and storing the hash value in a secondary hash set;

[0042] The restart task processing module is used to read the secondary hash set of the data comparison task when a data comparison restart task is identified, obtain the data segment that needs to be re-compared, and execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

[0043] A restart mass data comparison system, the system comprising at least one processor; and

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the restarting mass data comparison method.

[0046] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the restarting mass data comparison method.

[0047] Beneficial Effects : The embodiments of the present invention disclose a method, device, system and medium for restarting massive data comparison. Compared with the prior art, the present invention improves the efficiency of data comparison by adopting a segmented comparison method. Moreover, when the data comparison process needs to be re-compared, the data segment that needs to be re-compared can be quickly located by reading the corresponding hash value, thereby enhancing the convenience of data comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0049] Figure 1 A flowchart of a method for restarting mass data comparison provided by an embodiment of the present invention;

[0050] Figure 2 Another flow chart of the method for restarting massive data comparison provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of functional modules of a device for restarting mass data comparison provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the hardware structure of a system for restarting massive data comparison provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for restarting massive data comparison provided by the present invention, and the method specifically comprises the following steps:

[0055] S100, obtaining data tables that need to be compared based on the source database and the target database;

[0056] S200, splitting the data table into a plurality of data segments, and generating a query statement for each data segment;

[0057] S300, executing the query statement of the data segment to compare the data segment and generate difference data;

[0058] S400 performs secondary hash processing on the query statement of the data segment that has completed data comparison, obtains a hash value and stores the hash value in a secondary hash set;

[0059] S500: When a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data.

[0060] In this embodiment, the data table that needs to be compared is determined based on the source database and the target database. After the data table that needs to be compared is determined, the data table is split, the data table is divided into several data segments, and a query statement is generated for each data segment. The query statement is executed on the data segment to perform data comparison, and the difference data is output. The query statement of the data segment that has completed the data comparison is hashed, and finally a hash value is obtained, and the obtained hash value is stored in the secondary hash combination. When the data comparison task fails and needs to be reconnected, the system will read the secondary hash set of the data comparison task, and identify the data comparison restart task according to the hashcode (hash value) read in the secondary hash set. When the data comparison restart task is identified, the data position that needs to be reconnected is determined, and the data table that needs to be reconnected is obtained. The data segment is compared and the difference data is output. The system will mark all data segments that need to be reconnected and compared, and prepare for subsequent comparison operations until the data comparison tasks of all data tables are completed. The system can accurately identify which data segments need to be re-matched, thereby effectively resuming the matching process, avoiding duplication of work, and ensuring data consistency and integrity.

[0061] In one embodiment, the secondary hash set of the reading data comparison task, when identifying the data comparison restart task, obtains the data table that needs to be re-compared, including:

[0062] Read the secondary hash set of the data comparison task, and determine whether the secondary hash set is an empty set;

[0063] When the secondary hash set is an empty set, the data comparison task is started for the first time, and the data table to be compared is obtained;

[0064] When the secondary hash set is not an empty set, the data comparison task is restarted to obtain the data table that needs to be compared again.

[0065] In this embodiment, before the data comparison task starts, the system needs to read the secondary hash set (Map set), which contains the information of the data segments that have been compared before. The data structure of the secondary hash set is a Map, in which the key is the source table name, the value is a List, and the elements in the List are the hashCode (hash value) of the corresponding segment. The system needs to check whether this Map set is empty. This check is to determine whether the current data comparison task is started for the first time or restarted due to some reasons (such as power outage, network disconnection, server downtime, etc.). If the Map set is empty, it means that this is the first start of the data comparison task. In this case, the system will obtain all data tables that need to be compared and prepare to start the comparison process. If the Map set is not empty, it means that the data comparison task is restarted for some reason. The system needs to read the hashcode of the data segment that has been compared into the memory, obtain which data segments have completed the data comparison, which data segments have not completed the data comparison, and obtain those data tables that need to be re-compared.

[0066] Specifically, the Map collection is used to persist the comparison status of each data segment. By storing the hashCode of each segment, the system can quickly determine which segments have been compared when restarting, thus avoiding duplication of work. Whether it is the first startup or restart, the system needs to obtain the data tables that need to be compared. When starting for the first time, you may need to obtain all related data tables; when restarting, you only need to obtain those data tables that have not been compared.

[0067] Furthermore, the query SQL statement of each data segment generates a unique hashCode through secondary hashing, and this hashCode corresponds to the segment one by one. Therefore, even after restarting, as long as the query SQL remains unchanged, the hashCode of each segment will remain unchanged, allowing the system to accurately identify which segments have completed the comparison. Through this restart and continuation mechanism, the system can save a lot of time and resources wasted due to restart. Especially when processing massive data, this mechanism can significantly improve the efficiency and reliability of data comparison.

[0068] In one embodiment, splitting the data table into a plurality of data segments and generating a query statement for each data segment includes:

[0069] Creating a histogram in a source database or a target database according to the data table;

[0070] Splitting the data table into a specified number of data segments;

[0071] Generate query statements for the source database and the target database for each data segment based on the statistical information of the histogram.

[0072] In this embodiment, in the source database or the target database, creating a histogram for the data table to be compared is an effective data processing method. As a statistical tool, the histogram is used to display the distribution of data by dividing the data into several continuous intervals (buckets) and counting the number of data entries in each interval. Based on the statistical information of the histogram, the data table can be split into a specified number of data segments, each data segment corresponding to a bucket in the histogram.

[0073] Specifically, the histogram provides an intuitive view of data distribution and helps determine the boundaries of each bucket. In the data comparison task, the data table is split into 1024 buckets by default, but this setting can be adjusted according to the actual data volume and comparison requirements. The size of each bucket contains at least 10,000 data to ensure that the data volume of the task is moderate, avoiding too frequent task creation or a single task processing too much data.

[0074] For each data segment, a corresponding query statement is generated, which is used to retrieve the corresponding data segment from the source and target databases. The query statement contains necessary sorting instructions to ensure that the retrieved data is in the same order to facilitate comparison operations.

[0075] Once a task is created, it will be assigned to the comparison thread pool, which will be responsible for parallel execution. This parallel processing mechanism significantly improves the speed of data comparison, especially when processing large-scale data sets. The thread pool can reasonably allocate resources according to the number of tasks and system load, thereby improving the utilization of system resources.

[0076] Although there are many comparison tools on the market, such as SQL Compare, Navicat, SQL Plus, etc., these tools are already quite mature in performance. However, they usually use single-threaded comparison when processing a single table, which is still acceptable when the amount of data is small. But in the current environment of explosive growth in data volume, these tools appear to be inefficient when processing large-scale data. The component proposed in the present invention can perform multi-threaded comparison on the same table, significantly improving the efficiency of data processing.

[0077] In one embodiment, performing secondary hash processing on the query statement of the data segment that has completed the data comparison to obtain a hash value and storing the hash value in a secondary hash set includes:

[0078] Perform the first hash processing on the source database query statement of each data segment to obtain the first hash value;

[0079] Performing a second hash process on the first hash value to shorten the length of the first hash value and obtain a second hash value;

[0080] The second hash value is stored in a secondary hash set.

[0081] In this embodiment, the data comparer plays a vital role in the entire comparison process, and is responsible for executing all query and comparison operations. The design of this component is based on the thread pool mode, which ensures that each comparison task runs in an independent thread, thereby improving the overall execution efficiency. The differences found during the comparison process are sent to the blocking queue, and then the difference processor is responsible for processing these data.

[0082] In order to improve the efficiency of the data comparison process and ensure seamless continuation of previous tasks after the system restarts, a strategy was implemented to perform a secondary hash calculation on the query SQL before starting the task. This approach helps us quickly identify and skip completed comparison tasks.

[0083] The system first performs a SHA-256 hash operation on the query SQL in each task, which generates a 256-bit hash value as the initial identity of the query SQL. Subsequently, the system performs another hash operation on this initial hash value, and the result of this operation is a long integer number, which is regarded as the secondary hash code of the SQL.

[0084] Thanks to the orderliness and stability of the tasks created by the histogram, the query SQL of the new task will remain consistent and unchanged after each system restart. This means that the query statement for each data segment will have a unique and fixed hash code. In this way, we can ensure the uniqueness and traceability of the comparison task, so as to accurately determine whether the task needs to be re-executed after the restart.

[0085] In one embodiment, when a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data, including:

[0086] When the secondary hash set is not an empty set, the data comparison task is restarted, and the data comparison restart task is identified;

[0087] Read the secondary hash set of the data comparison task, and determine whether the second hash value exists in the secondary hash set;

[0088] When the second hash value exists in the secondary hash set, the data segment corresponding to the second hash value has completed the data comparison task;

[0089] When the second hash value does not exist in the secondary hash set, the data segment corresponding to the second hash value has not completed the data comparison task, and the data segment that needs to be compared again is obtained;

[0090] Execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

[0091] In this embodiment, when executing the data comparison task, the system uses the second hash value to determine the processing status of the data segment. By checking whether this hash value exists in a specific second hash set, the system can quickly determine whether the data segment has completed the comparison.

[0092] If the second hash value is already in the set, the corresponding data segment is considered to have completed the comparison task. In this case, the system does not need to perform any further comparison operations on the data segment.

[0093] On the contrary, if the second hash value does not appear in the set, it means that the corresponding data segment has not been compared. At this time, the system will perform a comparison operation on the data segment to find and record any existing differences.

[0094] In addition, the system will also check the hashCode set to confirm whether the hashCode of the current data segment has been processed. If the hashCode appears in the set, it means that the comparison of the data segment has been completed, so the system will skip the subsequent comparison process to avoid duplication of work.

[0095] In one embodiment, executing the query statement of the data segment to be re-compared to obtain the difference data specifically includes:

[0096] Execute query statements on the data segments of unfinished comparison tasks to perform data comparison;

[0097] The difference data generated by the data comparison is sent to the blocking queue for storage until the data comparison task of the data segment is completed.

[0098] In this embodiment, when processing unfinished data segments, the data comparer executes specific query SQL statements, which are designed to extract corresponding data segments from the source and target databases. In this way, the data comparer can obtain the precise data that needs to be compared, and the retrieved data then enters the data comparison stage, and the data comparer compares the source and target data line by line. The core goal of this process is to discover and mark any inconsistencies in the same data segments in the two databases.

[0099] During the comparison process, any data record that does not match the source and target will be identified as difference data. These difference data not only contain inconsistent field information, but also record the specific inconsistencies in detail. In order to ensure the integrity and order of the difference data and achieve efficient asynchronous processing, the difference data is sent to a blocking queue. The design of the blocking queue ensures that when the queue is full, any attempt to add a new element will wait until there is available space in the queue; similarly, when the queue is empty, attempts to remove elements will also wait until new elements are added to the queue. The strategy of using blocking queues to store and process difference data not only optimizes the efficiency of data processing, but also improves the responsiveness of the system. This process ensures that the system can continuously and reliably manage difference data when processing large-scale data comparison tasks, thereby improving the overall data comparison quality.

[0100] In one embodiment, the step of sending the difference data generated by the data comparison to a blocking queue for storage until the data comparison task of the data segment is completed further includes:

[0101] After the data comparison task is completed, the second hash value corresponding to the data segment is stored in the secondary hash set;

[0102] Determine whether the data comparison task of the data table is completed according to the second hash value in the second hash set;

[0103] When the data comparison task of the data table is completed, the difference data is extracted from the blocking queue, the difference data is summarized and processed, and a data report is generated from the summarized difference data and transmitted to a designated location for storage.

[0104] In this embodiment, when the comparison task of a data segment is successfully completed, the system will store the secondary hash value of the data segment in a secondary hash set. This set uses the data table name as the key and its value is a list containing the secondary hash values ​​of all completed data segments.

[0105] By checking the secondary hash values ​​in the secondary hash set, the system can determine whether the comparison task of a specific data table has been completed. If the secondary hash values ​​of all relevant data segments of a data table exist in the set, then it can be considered that the comparison work of the data table has been completed.

[0106] After confirming that the comparison task of a data table is completed, the system will extract all the difference data of the data table from the blocking queue. This data will then be summarized, including merging duplicates, sorting and organizing for further analysis and report generation.

[0107] The aggregated difference data is used to create a detailed data comparison report that documents all differences found during the comparison process. This report is then transmitted and stored in a designated location, such as a file system, database, or other data storage system.

[0108] In this way, the system can effectively manage and record the status of data comparison to ensure data consistency and integrity. At the same time, it provides users or system administrators with detailed data comparison result reports, which can be used for subsequent data analysis, review or record retention.

[0109] Overall, this design not only improves the efficiency and response speed of data processing, but also enhances the reliability and quality of data comparison, bringing significant optimization to the data processing process.

[0110] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of the method for restarting the massive data comparison provided by the present invention, and the method specifically comprises the following steps:

[0111] (1) The comparison process starts;

[0112] (2) Reading completed segment information from the completed segment set;

[0113] (3) Task splitting;

[0114] (4) Multi-threaded processing of multiple segments;

[0115] (5) Perform secondary hashing on the task query statement;

[0116] (6) Determine whether the segment has been compared. If the segment has been compared, proceed to (7). If the segment has not been compared, proceed to (4).

[0117] (7) Complete data comparison.

[0118] Before data comparison, the present invention needs to determine the data tables that need to be compared in the source database and the target database. After determining the data tables that need to be compared, the system will read the secondary hash set of the data comparison task, identify the data comparison restart task according to the Hash code in the secondary hash set, and when the data comparison restart task is identified, determine the data position that needs to be reconnected and compared, and obtain the data table that needs to be reconnected. Then, the data table is split, the data table is divided into several data segments, and a query statement is generated for each data segment, and the query statement of the data segment is processed by hash value, and finally the hash value is obtained. It is judged whether the obtained hash value exists in the secondary hash set. If the hash value is not in the secondary hash set, the data segment has not completed the data comparison task, and the data segment is compared to output the difference data. The system will mark all the data segments that need to be reconnected and compared, and prepare for subsequent comparison operations until the data comparison tasks of all data tables are completed. The system can accurately identify which data segments need to be reconnected, thereby effectively restoring the comparison process, avoiding duplication of work, and ensuring the consistency and integrity of the data.

[0119] A device for restarting mass data comparison, the device comprising:

[0120] A data table module 11 is used to obtain a data table that needs to be compared based on a source database and a target database;

[0121] A data splitting module 12 is used to split the data table into a plurality of data segments and generate a query statement for each data segment;

[0122] A data comparison module 13, used for executing the query statement of the data segment to compare the data segment and generate difference data;

[0123] A hash processing module 14 is used to perform secondary hash processing on the query statement of the data segment that has completed the data comparison, obtain a hash value and store the hash value in a secondary hash set;

[0124] The restart task processing module 15 is used to read the secondary hash set of the data comparison task when a data comparison restart task is identified, obtain the data segment that needs to be re-compared, and execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

[0125] like Figure 3As shown, the data table module 11, the data splitting module 12, the data comparison module 13, the hash processing module 14 and the restart task processing module 15 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for restarting the execution process of the massive data comparison method than the program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0126] In one embodiment, the reading task identification module includes:

[0127] A secondary hash set reading unit, used to read the secondary hash set of the data comparison task and determine whether the secondary hash set is an empty set;

[0128] A task initial startup unit, used for, when the secondary hash set is an empty set, the data comparison task is initially started, and obtaining a data table to be compared;

[0129] The task restarting unit is used to restart the data comparison task when the secondary hash set is not an empty set, and obtain the data table that needs to be compared again.

[0130] In one embodiment, the data splitting module 12 includes:

[0131] A histogram creation unit is used to create a histogram in a source-end database or a target-end database according to the data table;

[0132] A data table splitting unit, used for splitting the data table into a specified number of data segments;

[0133] A query statement unit is used to generate query statements of the source database and the target database for each data segment based on the statistical information of the histogram.

[0134] In one embodiment, the hash processing module 14 includes:

[0135] A first hash processing unit, used to perform a first hash processing on a source database query statement of each data segment to obtain a first hash value;

[0136] A second hash processing unit, used to perform a second hash processing on the first hash value, shorten the length of the first hash value, and obtain a second hash value;

[0137] The hash value storage unit is used to store the second hash value in a secondary hash set.

[0138] In one embodiment, the restart task processing module 15 includes:

[0139] A task identification unit is used to identify the data comparison restart task when the secondary hash set is not an empty set, then the data comparison task is restarted

[0140] A secondary hash set judgment unit, used to read the secondary hash set of the data comparison task and judge whether the second hash value exists in the secondary hash set;

[0141] A comparison task completion unit, used for, when the second hash value exists in the secondary hash set, the data segment corresponding to the second hash value has completed the data comparison task;

[0142] A data segment acquisition unit, configured to acquire a data segment that needs to be re-matched when the second hash value does not exist in the second hash set and the data segment corresponding to the second hash value has not completed the data comparison task;

[0143] The data segment comparison unit is used to execute the query statement of the data segment that needs to be re-compared to obtain difference data.

[0144] In one embodiment, the data comparison unit specifically includes:

[0145] Execute query statements on the data segments of unfinished comparison tasks to perform data comparison;

[0146] The difference data generated by the data comparison is sent to the blocking queue for storage until the data comparison task of the data segment is completed.

[0147] In one embodiment, the data processing module further includes:

[0148] A hash value storage unit, used for storing the second hash value corresponding to the data segment into a secondary hash set after the data comparison task is completed;

[0149] A comparison task judgment unit, used for judging whether the data comparison task of the data table is completed according to the second hash value in the second hash set;

[0150] The data summary unit is used to extract the difference data from the blocking queue when the data comparison task of the data table is completed, summarize the difference data, generate a data report from the summarized difference data, and transmit it to a designated location for storage.

[0151] Another embodiment of the present invention provides a restarting mass data comparison system, the system comprising at least one processor; and

[0152] a memory communicatively connected to the at least one processor; wherein,

[0153] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the restarting mass data comparison method.

[0154] One or more processors 110 and memory 120, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0155] The processor 110 is used to complete various control logics of the control system 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0156] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the restarting massive data comparison method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the control system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implementing the restarting massive data comparison method in the above method embodiment.

[0157] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the control system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the control system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] One or more units are stored in the memory 120, and when executed by one or more processors 110, the restarting massive data comparison method in any of the above method embodiments is executed, for example, the above described Figure 1 The method comprises steps S100 to S500.

[0159] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S100 to S500.

[0160] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0161] In summary, the present invention discloses a method, device, system and medium for restarting massive data comparison, including: obtaining a data table that needs to be compared based on a source database and a target database; splitting the data table into several data segments, and generating corresponding query statements; executing the query statements to compare the data segments and generate difference data; performing secondary hash processing on the query statements of the data segments that have completed the data comparison, obtaining hash values ​​and storing the hash values ​​in secondary hash sets; when a data comparison restart task is identified, reading the secondary hash set of the data comparison task, obtaining the data segments that need to be re-compared; executing the query statements of the data segments that need to be re-compared, and obtaining difference data. The present invention improves the efficiency of data comparison by adopting a segmented comparison method, and when re-comparison is required during the data comparison process, the data segments that need to be re-compared can be located by reading the corresponding hash values, thereby enhancing the convenience of data comparison.

[0162] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0163] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A restart method for comparing massive data, characterized in that: The steps include: Obtain the data tables that need to be compared based on the source database and the target database; Splitting the data table into a plurality of data segments, and generating a query statement for each data segment; Execute the query statement of the data segment to compare the data segment and generate difference data; Performing secondary hash processing on the query statement of the data segment that has completed the data comparison, obtaining a hash value and storing the hash value in a secondary hash set; When a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data.

2. The restarting mass data comparison method according to claim 1 is characterized in that: When a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data, including: Read the secondary hash set of the data comparison task, and determine whether the secondary hash set is an empty set; When the secondary hash set is an empty set, the data comparison task is started for the first time; When the secondary hash set is not an empty set, the data comparison task is restarted.

3. The restarting massive data comparison method according to claim 1 is characterized in that: The step of splitting the data table into a plurality of data segments and generating a query statement for each data segment includes: Creating a histogram in a source database or a target database according to the data table; Splitting the data table into a specified number of data segments; Generate query statements for the source database and the target database for each data segment based on the statistical information of the histogram.

4. The restarting mass data comparison method according to claim 1 is characterized in that: The performing secondary hash processing on the query statement of the data segment that has completed the data comparison, obtaining a hash value and storing the hash value in the secondary hash set, comprises: Perform the first hash processing on the source database query statement of each data segment to obtain the first hash value; Performing a second hash process on the first hash value to shorten the length of the first hash value and obtain a second hash value; The second hash value is stored in a secondary hash set.

5. The restarting mass data comparison method according to claim 4 is characterized in that: When a data comparison restart task is identified, the secondary hash set of the data comparison task is read to obtain the data segment that needs to be re-compared, and the query statement of the data segment that needs to be re-compared is executed to obtain the difference data, including: When the secondary hash set is not an empty set, the data comparison task is restarted, and the data comparison restart task is identified; Read the secondary hash set of the data comparison task, and determine whether the second hash value exists in the secondary hash set; When the second hash value exists in the secondary hash set, the data segment corresponding to the second hash value has completed the data comparison task; When the second hash value does not exist in the secondary hash set, the data segment corresponding to the second hash value has not completed the data comparison task, and the data segment that needs to be compared again is obtained; Execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

6. The restarting mass data comparison method according to claim 5 is characterized in that: The executing the query statement of the data segment to be re-compared to obtain the difference data specifically includes: Execute query statements on the data segments of unfinished comparison tasks to perform data comparison; The difference data generated by the data comparison is sent to the blocking queue for storage until the data comparison task of the data segment is completed.

7. The restarting mass data comparison method according to claim 6 is characterized in that: The step of sending the difference data generated by the data comparison to a blocking queue for storage until the data comparison task of the data segment is completed further includes: After the data comparison task is completed, the second hash value corresponding to the data segment is stored in the secondary hash set; Determine whether the data comparison task of the data table is completed according to the second hash value in the second hash set; When the data comparison task of the data table is completed, the difference data is extracted from the blocking queue, the difference data is summarized and processed, and a data report is generated from the summarized difference data and transmitted to a designated location for storage.

8. A restarting mass data comparison device, characterized in that: The device comprises: The data table module is used to obtain the data table that needs to be compared based on the source database and the target database; A data splitting module, used to split the data table into several data segments and generate a query statement for each data segment; A data comparison module, used for executing the query statement of the data segment to compare the data segment and generate difference data; A hash processing module, used for performing secondary hash processing on the query statement of the data segment that has completed the data comparison, obtaining a hash value and storing the hash value in a secondary hash set; The restart task processing module is used to read the secondary hash set of the data comparison task when a data comparison restart task is identified, obtain the data segment that needs to be re-compared, and execute the query statement of the data segment that needs to be re-compared to obtain the difference data.

9. A restart system for comparing massive data, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the restarting massive data comparison method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the restarting massive data comparison method described in any one of claims 1-7.

Images