Data comparison method and device
Through the data comparison method with fixed-length step, the problem of low data comparison efficiency in the data aggregation and synchronization process in big data applications is solved, efficient and accurate comparison under limited memory resources is achieved, and data synchronization and matching are improved. The efficiency of the account.
Patent Information
- Application Number
- CN202411056189.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-15
AI Technical Summary
In big data applications, during the data aggregation and synchronization process, it is difficult for the existing technology to achieve efficient and accurate data comparison, especially when server resources are limited, when the data volume is large and multi-task concurrently run, it is difficult to quickly and accurately Complete data comparison and difference analysis.
Using a fixed-length stepwise data comparison method, through a preset fixed data step length, data is collected from the sorted data in sequence, the data set in memory is updated, and the intersection and difference set comparison method is used. Determine the intersection and difference set of data, so as to conduct large-scale data comparison under finite memory resources.
It realizes efficient and accurate data comparison under limited memory resources, saves memory resources, can handle the comparison of large-scale ordered data sets, and improves the efficiency of data synchronization and reconciliation.
Smart Images

Figure CN118964342A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present invention relate to big data technology, and more particularly, to a data comparison method and device. Background Art
[0002] With the development of science and technology, big data is needed in all walks of life. Big data technology is so important that its data collection, synchronization, storage, search, sharing, analysis, and even visual presentation have become important research topics at present, and data synchronization is a very important part of the construction of the big data application base. In the process of many project development and implementation, customers and R&D personnel often pay more attention to data governance, mining and analysis applications, but later they find that data aggregation may be the biggest difficulty and the most time-consuming and labor-intensive work. Being able to extract external data to the data platform in a timely, fast, accurate and complete manner is a problem faced by many developers.
[0003] During the data aggregation and synchronization process, in order to ensure that the target data and the source data are completely consistent, it is necessary to identify the addition, modification, and deletion operations of the source data. In order to achieve the above purpose, the source data needs to be compared with the target data. In addition, after the data aggregation and synchronization is completed, data reconciliation is required to check again whether the target data and the source data are consistent. If they are inconsistent, the difference data needs to be found and further complement operations need to be performed. In this case, there is also a need for data comparison between the data. In the implementation of the data project, considering that the amount of data in the table is continuously increasing, and there will be multiple such data synchronization tasks running concurrently, it is necessary to consider the problem of server resource limitations.
[0004] Therefore, a more effective method for comparing source data with target data is needed. Summary of the invention
[0005] One or more embodiments of the present invention describe a data comparison method and device, which can complete data comparison more efficiently and accurately.
[0006] According to a first aspect, a data comparison method is provided, the method comprising:
[0007] Obtain at least two sorted source data; wherein the source data is data from a data source;
[0008] Acquire at least two sorted target data; wherein the target data is the data stored in the updated party;
[0009] Set the source dataset and target dataset;
[0010] Whenever the source data set is empty, at least two source data are collected in sequence from all the sorted source data according to a preset fixed data step length, and the source data set set in the memory is updated using the ordered source data collected this time;
[0011] Whenever the target data set is empty, at least two target data are collected from all the sorted target data in sequence according to a preset fixed data step length, and the target data set set in the memory is updated using the ordered target data collected this time;
[0012] Whenever at least one of the source data set and the target data set is updated, the following steps are performed in sequence: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set. If so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from both the source data set and the target data set; then, for each data currently remaining in the source data set, determine one by one whether the data meets the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set; finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set, and delete each data currently remaining in the target data set.
[0013] The data is: text type data, digital type data or character type data;
[0014] and / or,
[0015] When the data is text type data or character type data, the sorting includes: sorting according to a dictionary sorting method;
[0016] and / or,
[0017] The fixed data step length is 10;
[0018] and / or,
[0019] The at least two source data include: full data from the data source or incremental data from the data source.
[0020] When it is determined that a piece of data belongs to the intersection of the source data set and the target data set, the method further includes: modifying the value of the data in the target data to the value of the data in the source data.
[0021] When it is determined that a data in the source data set belongs to a difference set of the source data set with respect to the target data set, the method further includes: adding the data to the target data.
[0022] When it is determined that each data currently remaining in the target data set belongs to a difference set of the target data set with respect to the source data set, the method further includes: deleting each data currently remaining in the target data set from the target data.
[0023] The method of determining, for each data currently remaining in the source data set, whether the data satisfies a preset difference set deletion condition, includes: determining, for each data currently remaining in the source data set, whether the data is less than the last data currently remaining in the target data set; if so, determining that the data belongs to a difference set of the source data with respect to the target data, and determining that the data satisfies the preset difference set deletion condition; and if so, determining that the data does not currently satisfy the difference set deletion condition.
[0024] According to a second aspect, a data comparison device is provided, the device comprising:
[0025] A source data receiving module is configured to obtain at least two source data after sorting; wherein the source data is data from a data source;
[0026] A target data acquisition module, configured to acquire at least two sorted target data; wherein the target data is the data stored in the updated party;
[0027] The data set processing module is configured to set a source data set and a target data set; whenever the source data set is empty, at least two source data are sequentially collected from all the sorted source data according to a preset fixed data step length, and the source data set in the memory is updated with the ordered source data collected this time; whenever the target data set is empty, at least two target data are sequentially collected from all the sorted target data according to a preset fixed data step length, and the target data set in the memory is updated with the ordered target data collected this time;
[0028] The comparison processing module is configured to execute in sequence whenever at least one of the source data set and the target data set is updated: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set. If so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from the source data set and the target data set; then, for each data currently remaining in the source data set, determine one by one whether the data meets the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set; finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set.
[0029] The device further comprises: a synchronization processing module;
[0030] The sync processing module is configured to perform:
[0031] After the comparison processing module determines that a piece of data belongs to the intersection of the source data set and the target data set, the synchronization processing module modifies the value of the data in the target data to the value of the data in the source data.
[0032] and / or,
[0033] After the comparison processing module determines that a data in the source data set belongs to a difference set of the source data set with respect to the target data set, the synchronization processing module adds the data to the target data set;
[0034] and / or,
[0035] After the comparison processing module determines that each data currently remaining in the target data set belongs to a difference set of the target data set with respect to the source data set, the synchronization processing module deletes each data currently remaining in the target data set from the target data.
[0036] When the comparison processing module determines whether the data satisfies the preset difference set deletion condition one by one, it is configured to execute:
[0037] For each data currently remaining in the source data set, determine one by one whether the data is less than the last data currently remaining in the target data set. If it is less, determine that the data belongs to the difference set of the source data for the target data, and determine that the data meets the preset difference set deletion condition. If it is greater, determine that the data currently does not meet the difference set deletion condition.
[0038] According to a third aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any embodiment of the present invention is implemented.
[0039] The data comparison method and device provided by the embodiment of the present invention adopt a fixed-length stepping method to perform data comparison, that is, each time a fixed number of data, such as 10 data, are taken out for comparison according to a fixed data stepping length, such as 10 data, and it is possible to determine whether a data, such as data M, belongs to the intersection of a source data set and a target data set, that is, whether the data M of the source party already exists in the target data of the updated party; after performing the comparison of the intersection, the comparison of the source data set with the target data set can be performed, that is, to determine whether a data, such as data N, belongs to the difference set of the source data set with respect to the target data set, that is, whether the data N already exists in the source data but does not exist in the target data; after performing the comparison of the source data set with the difference set of the target data set, the comparison of the target data set with the difference set of the source data set can be performed, that is, to determine whether a data, such as data K, belongs to the difference set of the target data set with respect to the source data set, that is, whether the data K already exists in the target data but does not exist in the source data. It can be seen that the method of the embodiment of the present invention, through multiple batches of comparisons with each batch having a small amount of data, adopts a specific intersection and difference comparison method, and can accurately obtain various comparison results of two data under limited memory resources.
[0040] Furthermore, in an embodiment of the present invention, the length of the data read into the memory can be controlled by setting the data step length, that is, the length of the source data set and the target data set in the memory can be controlled, so that data comparison of two ordered data sets with very large data volumes can be achieved under limited memory resources, thereby greatly saving memory resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 It is a flow chart of a data comparison method in one embodiment of the present invention.
[0043] Figure 2 It is a flow chart of a data comparison method in another embodiment of the present invention.
[0044] Figure 3 It is a schematic diagram of the structure of a data comparison device in one embodiment of the present invention.
[0045] Figure 4It is a schematic diagram of the structure of a data comparison device in another embodiment of the present invention. DETAILED DESCRIPTION
[0046] In actual business implementation, it is often necessary to compare data between two systems, so that subsequent data processing can be performed based on the comparison results. For example, the public security department counts the information of new births and new deaths every year, while the national finance department needs the latest population information. Therefore, it is necessary to compare the population data of the public security system with the population data of the finance department, so that the corresponding population data can be updated in the finance department based on the comparison results. With the continuous increase in the amount of information, the current data comparison is usually of a very large order of magnitude, so a more efficient data comparison method is needed.
[0047] The solution provided by the present invention is described below in conjunction with the accompanying drawings.
[0048] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.
[0049] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0050] Figure 1 1 is a flow chart of a data comparison method in one embodiment of the present invention. The execution subject of the method is a data comparison device. Figure 1 , the method comprising:
[0051] Step 101: Obtain at least two sorted source data; wherein the source data is data from the data source.
[0052] Step 105: Obtain at least two sorted target data; wherein the target data is the data stored in the updated party.
[0053] Step 107: Set the source data set and the target data set.
[0054] Step 109: Whenever the source data set is empty, each data is collected in sequence from all the sorted source data according to the preset fixed data step length, and the source data set in the memory is updated with the ordered data collected this time.
[0055] Step 111: Whenever the target data set is empty, each data is collected in sequence from all the sorted target data according to a preset fixed data step length, and the target data set in the memory is updated with the ordered data collected this time.
[0056] Step 113: Whenever at least one of the source data set and the target data set is updated, execute in sequence: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set; if so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from the source data set and the target data set.
[0057] Step 115: Afterwards, for each data currently remaining in the source data set, determine one by one whether the data satisfies the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set.
[0058] Step 117: Finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that the data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set, and delete the data currently remaining in the target data set.
[0059] according to Figure 1 It can be seen from the process shown that in the embodiment of the present invention, a fixed-length step method is used for data comparison, that is, each time a fixed data step length, such as 10 data, is used to take out a group of data of a fixed number, such as 10 data, for comparison, and it can be determined whether a data, such as data M, belongs to the intersection of the source data set and the target data set, that is, whether the data M of the source party already exists in the target data of the updated party; after the intersection comparison is performed, the source data set can be compared with the difference set of the target data set, that is, it is determined whether a data, such as data N, belongs to the difference set of the source data set with respect to the target data set, that is, whether the data N already exists in the source data but does not exist in the target data; after the source data set is compared with the difference set of the target data set, the target data set can be compared with the difference set of the source data set, that is, it is determined whether a data, such as data K, belongs to the difference set of the target data set with respect to the source data set, that is, whether the data K already exists in the target data but does not exist in the source data. It can be seen that the method of the embodiment of the present invention, through multiple batches of comparisons with each batch having a small amount of data, adopts a specific intersection and difference comparison method, and can accurately obtain various comparison results of two data under limited memory resources.
[0060] And, according to Figure 1It can be seen from the process shown that in an embodiment of the present invention, the length of the data read into the memory can be controlled by setting the data step length, that is, the length of the source data set and the target data set in the memory can be controlled, so that data comparison of two ordered data sets with very large data levels can be achieved under limited memory resources, thereby greatly saving memory resources.
[0061] Below Figure 1 Each of the steps shown is explained separately.
[0062] First, for step 101: obtain at least two source data; wherein the source data is the data from the data source.
[0063] As mentioned above, in actual business implementation, there is often a need to compare data between two systems, so that subsequent corresponding data processing can be performed based on the comparison results. For example, the public security department will count the information of new births and new deaths every year, and the national finance department needs the latest population information. Therefore, it is necessary to compare the population data of the public security system with the population data of the finance department, so as to update the corresponding population data in the finance department based on the comparison results. In this example, the data source is the system of the public security department, and the updated party is the finance department.
[0064] In this step 101, the acquired data may be the full amount of data from the data source.
[0065] In this step 101, the data obtained may also be the incremental data of the data source. In the case of incremental data, in order to reduce the number of data comparisons, only the changed data of the data source, such as the newly added data, the deleted data, and the modified data, are notified to the updated party, so that the updated party completes the data synchronization after the data comparison. For example, in one implementation, in this step 101, the source data is generated using the incremental data of the data source. For example, the source data includes: data of all new population increases, data of all population decreases, and data of all population changes in the past year.
[0066] Next, for step 105: obtain at least two sorted target data.
[0067] In the embodiment of the present invention, in order to complete the data comparison more quickly and simply, the data of the two parties can be sorted and the sorted data of the two parties can be compared. The sorting method can be: sorting from small to large; or sorting in order of precedence.
[0068] In the embodiment of the present invention, the data to be compared may be text type data, number type data or character type data.
[0069] When the data is of numeric type, you can directly sort it according to the size of the number.
[0070] When the data is text type data or character type data, the sorting method includes: sorting according to the dictionary sorting method. For example, if the data is Chinese characters such as names, each data can be sorted according to the order in which the data appears in the Chinese dictionary. For another example, if the data is English letters, each data can be sorted according to the order in which the letters appear in the alphabet.
[0071] For example, in a data synchronization scenario, the source of data is mostly relational databases or big data platforms. The sorting operation of the source data can be performed directly in the database, and the data output by the database is ordered data.
[0072] Step 107: Set the source data set and the target data set.
[0073] In actual business implementation, the amount of data on both sides that need to be compared is usually very huge. In an embodiment of the present invention, in order to achieve data comparison of two ordered data sets with very large data volumes under limited memory resources, a source data set and a target data set can be set. In this way, the source data set and the target data set can be used to achieve batch data comparison, and only the current data in the source data set and the target data set in the memory are compared each time.
[0074] Next, for step 109: whenever the source data set is empty, each data is collected in sequence from all the sorted source data according to the preset fixed data step length, and the source data set in the memory is updated with the ordered data collected this time.
[0075] The processing of step 109 can be performed when the data is initially read and when the source data set is cleared during the comparison process. For example, if the preset fixed data step length is 10 and there are 8 million source data in total, in step 109, 10 data are collected in sequence each time from the 8 million sorted source data and put into the source data set.
[0076] Next, for step 111: whenever the target data set is empty, each data is collected in sequence from all the sorted target data according to the preset fixed data step length, and the target data set in the memory is updated using the ordered data collected this time.
[0077] The processing of step 111 can be performed when the data is initially read and when the target data set is cleared during the comparison process. For example, if the preset fixed data step length is 10 and there are 200 million target data in total, in step 111, 10 data are collected in order each time from the sorted 200 million target data and put into the target data set.
[0078] For example, the number of batches of data acquired (i.e., the fixed step length used for data comparison each time) can be controlled through cursor streaming reading to reduce the client memory resource usage used for data comparison.
[0079] The following takes the data step length of 10 each time as an example (ie, 10 pieces of source data and 10 pieces of target data are obtained each time).
[0080] As shown in the following table, there are two data sets, each with a capacity of 10 data items. The data value in the set represents the unique identifier of each data item.
[0081] Source dataset A:
[0082] Table 1
[0083] Target dataset B:
[0084] 1 3 4 5 6 7 8 9 12 13
[0085] Table 2
[0086] For example, through the processing of step 109 and step 111 , 10 data as shown in Table 1 are put into the source data set A, and 10 data as shown in Table 2 are put into the target data set B.
[0087] Next, for step 113: Whenever at least one of the source data set and the target data set is updated, execute in sequence: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set; if so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from the source data set and the target data set.
[0088] For example, referring to Table 1 and Table 2 above, the 10 data in the source data set A are compared one by one with the 10 data in the target data set B. It is found that the data 1, 3, 4, 5, 6 and 7 shown in Table 1, a total of 6 data are the same as the 6 data in the target data set B. This means that 1, 3, 4, 5, 6 and 7, a total of 6 data, belong to the intersection of the source data set and the target data set. The 6 data can be put into the route of A∩B and deleted from the source data set A and the target data set B.
[0089] Subsequently, the corresponding operation can be performed on the target data of the updated party for the data placed in the route of A∩B. For the data in the intersection, since the data of the source party already exists in the target data of the updated party, it means that the data values of each data in the intersection in the target data need to be updated, that is, the value of the data in the target data is modified to the value of the data in the source data.
[0090] After the processing of step 113, the current source data set A is as shown in Table 3 below:
[0091] Table 3
[0092] After the processing of step 113, the current target data set B is as shown in Table 4 below:
[0093] Target dataset B:
[0094] 8 9 12 13
[0095] Table 4
[0096] Next, for step 115: thereafter, for each data currently remaining in the source data set, determine one by one whether the data satisfies the preset difference set deletion condition. If so, delete the data from the source data set. If not, retain the data in the source data set.
[0097] When executing this step 115, none of the currently remaining data in the source data set A belongs to the current target data set B. However, it cannot be determined that all the currently remaining data in the source data set A must be the difference set of the source data relative to the target data. Because the source data and the target data are compared in batches, if there is data in the target data that has not been read into the target data set and has not participated in the comparison, it cannot be determined that all the currently remaining data in the source data set A is the difference set of the source data relative to the target data. Therefore, the present invention pre-sets a difference set deletion condition. Only when the remaining data meets the difference set deletion condition can it be determined that the data is the difference set of the source data relative to the target data.
[0098] In one embodiment of the present invention, the implementation process of this step 115 includes: for each data currently remaining in the source data set, determine one by one whether the data is less than the last data currently remaining in the target data set; if it is less than, determine that the data belongs to the difference set of the source data with respect to the target data, that is, determine that the data meets the preset difference set deletion condition, and then delete the data from the source data set; if it is greater than, determine that the data does not currently meet the difference set deletion condition, and then retain the data in the source data set.
[0099] Because in the target data set B, the data are arranged in ascending order or in order of precedence, therefore, it is determined whether each data currently remaining in the source data set is less than the last data currently remaining in the target data set, that is, whether each data currently remaining in the source data set A is less than the maximum value in the target data set B. If less than, it is identified that this data belongs to the difference set of the source data relative to the target data; this data is routed to the route of AB, and this data is deleted from the set A. For example, it is determined that the two data 2 and 10 shown in Table 3 are less than the maximum value 13 in the target data set B shown in Table 4, and belong to the difference set of the source data relative to the target data, and the data 2 and 10 are routed to the route of AB, and the two data are deleted from the source data set A.
[0100] For 15 and 16 in the source data set A shown in Table 3, although 15 and 16 do not belong to the target data set B, because the comparison has not been completed, it is not possible to determine whether 15 and 16 belong to the difference set of the source data relative to the target data. Therefore, it is necessary to retain the data 15 and 16 in the source data set so that the retained data 15 and 16 can be compared with the next batch of data that will be newly read into the target data set B.
[0101] Subsequently, the corresponding operation processing can be performed on the target data of the updated party for the data placed in the route of AB. For the data in AB, if the data exists in the source party but does not exist in the target data of the updated party, it means that the data in AB needs to be added to the target data, that is, the data is added to the target data.
[0102] After the processing of step 115 , the current source data set A is as shown in Table 5 below, and the target data set B is still as shown in Table 4.
[0103] Source dataset A:
[0104] 15 16
[0105] Table 5
[0106] Target dataset B:
[0107] 8 9 12 13
[0108] Table 4
[0109] Finally, for step 117: determine whether the source data set and the target data set are currently empty. If both are not empty, determine that the data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set, and delete the data currently remaining in the target data set.
[0110] After processing in step 113 and step 115, if the amount of data in the two data sets is still not empty, the data in the target data set all belong to the difference set of B for A, and the data is routed to the route of BA, and the data is deleted from the set B. For example, referring to Table 4 and Table 5, all the data 8, 9, 12, and 13 currently remaining in the target data set B are the difference sets of B for A, that is, they exist in the target data set, but not in the source data set.
[0111] Thereafter, because the target data set B is empty, the process of step 111 may be performed, and steps 113 , 115 , and 117 may be performed again.
[0112] Subsequently, the corresponding operation processing can be performed on the target data of the updated party for the data in the route put into BA. For the data in BA, if the data exists in the target data of the updated party but does not exist in the source data of the source party, it means that the data in BA needs to be deleted from the target data, that is, the data is deleted from the target data. The data currently remaining in the target data set is deleted from the target data.
[0113] Through the above process, the source data is read into the source data set in batches, and the target data is read into the target data set in batches. Each batch is read according to the above process. Figure 1 By executing the process of the method shown, accurate data comparison can be achieved under limited memory resources.
[0114] Figure 2 FIG. 1 is a flow chart of a data comparison method in another embodiment of the present invention. Figure 1 and Figure 2 In a specific example, the method of comparing the source data of the source party with the target data of the updated party can be seen in Figure 2 The specific process shown.
[0115] It can be seen that an embodiment of the present invention provides a method for searching for differences in large amounts of data in a limited resource. Through the invention, specific difference data can be found in a relatively short time in limited hardware resources, inconsistent data can be found in a timely manner, and the stability and automation efficiency of the system can be improved; it can also provide support for subsequent data supplementation or abnormal data analysis.
[0116] The present invention can realize a method for comparing two ordered data sets of very large data volumes under limited memory resources. By comparing data using this method, the intersection of the source data set and the target data set, the difference of the source data set with respect to the target data set, and the difference of the target data set with respect to the source data set can be identified, and based on this, the type of data operation that needs to be performed in the target data can be identified: addition, update / empty operation, deletion.
[0117] Figure 3 FIG. 1 is a schematic diagram of the structure of a data comparison device in one embodiment of the present invention. Figure 3 , the device comprises:
[0118] The source data receiving module 301 is configured to obtain at least two source data after sorting; wherein the source data is the data of the data source party;
[0119] The target data acquisition module 303 is configured to acquire at least two sorted target data; wherein the target data is the data stored in the updated party;
[0120] The data set processing module 304 is configured to set a source data set and a target data set; whenever the source data set is empty, at least two source data are sequentially collected from all the sorted source data according to a preset fixed data step length, and the source data set in the memory is updated with the ordered source data collected this time; whenever the target data set is empty, at least two target data are sequentially collected from all the sorted target data according to a preset fixed data step length, and the target data set in the memory is updated with the ordered target data collected this time;
[0121] The comparison processing module 305 is configured to execute in sequence whenever at least one of the source data set and the target data set is updated: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set. If so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from the source data set and the target data set; then, for each data currently remaining in the source data set, determine one by one whether the data meets the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set; finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set.
[0122] See also Figure 4 , in one embodiment of the device of the present invention, the device further comprises: a synchronization processing module 401;
[0123] The synchronization processing module 401 is configured to execute:
[0124] After the comparison processing module 305 determines that a piece of data belongs to the intersection of the source data set and the target data set, the synchronization processing module 401 modifies the value of the data in the target data to the value of the data in the source data.
[0125] and / or,
[0126] After the comparison processing module 305 determines that a data in the source data set belongs to the difference set of the source data set with respect to the target data set, the synchronization processing module 401 adds the data to the target data set;
[0127] and / or,
[0128] After the comparison processing module 305 determines that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set, the synchronization processing module 401 deletes each data currently remaining in the target data set from the target data.
[0129] In one embodiment of the device of the present invention, the data is: text type data, number type data or character type data.
[0130] and / or,
[0131] When the data is text type data or character type data, the sorting includes: sorting according to a dictionary sorting method;
[0132] and / or,
[0133] The fixed data step length is 10;
[0134] and / or,
[0135] The at least two source data include: full data from the data source or incremental data from the data source.
[0136] In one embodiment of the device of the present invention, when the comparison processing module 305 determines whether the data satisfies the preset difference set deletion condition one by one, it is configured to execute:
[0137] For each data currently remaining in the source data set, determine one by one whether the data is less than the last data currently remaining in the target data set. If it is less, determine that the data belongs to the difference set of the source data for the target data, and determine that the data meets the preset difference set deletion condition. If it is greater, determine that the data currently does not meet the difference set deletion condition.
[0138] It should be noted that the above-mentioned devices are usually implemented on the server side, and can be set on independent servers respectively, or some or all of the devices can be combined and set on the same server. The server can be a single server or a server cluster composed of multiple servers. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. The above-mentioned devices can also be implemented on a computer terminal with strong computing power.
[0139] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute a method in any one of the embodiments in the specification.
[0140] An embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method in any embodiment of the specification is implemented.
[0141] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the device of the embodiment of the present invention. In other embodiments of the specification, the above-mentioned device may include more or fewer components than those shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0142] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0143] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0144] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A data comparison method, characterized in that: The method includes: Obtain at least two sorted source data; wherein the source data is data from a data source; Acquire at least two sorted target data; wherein the target data is the data stored in the updated party; Set the source dataset and target dataset; Whenever the source data set is empty, at least two source data are collected in sequence from all the sorted source data according to a preset fixed data step length, and the source data set set in the memory is updated using the ordered source data collected this time; Whenever the target data set is empty, at least two target data are collected from all the sorted target data in sequence according to a preset fixed data step length, and the target data set set in the memory is updated using the ordered target data collected this time; Whenever at least one of the source data set and the target data set is updated, the following steps are performed in sequence: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set. If so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from both the source data set and the target data set; then, for each data currently remaining in the source data set, determine one by one whether the data meets the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set; finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set, and delete each data currently remaining in the target data set.
2. The method according to claim 1, characterized in that The data is: text type data, digital type data or character type data; and / or, When the data is text type data or character type data, the sorting includes: sorting according to a dictionary sorting method; and / or, The fixed data step length is 10; and / or, The at least two source data include: full data from the data source or incremental data from the data source.
3. The method according to claim 1, characterized in that When it is determined that a piece of data belongs to the intersection of the source data set and the target data set, the method further includes: modifying the value of the data in the target data to the value of the data in the source data.
4. The method according to claim 1, characterized in that: When it is determined that a data in the source data set belongs to a difference set of the source data set with respect to the target data set, the method further includes: adding the data to the target data.
5. The method according to claim 1, characterized in that When it is determined that each data currently remaining in the target data set belongs to a difference set of the target data set with respect to the source data set, the method further includes: deleting each data currently remaining in the target data set from the target data.
6. The method according to claim 1, characterized in that The step of determining, for each remaining data in the source data set, whether the data satisfies a preset difference set deletion condition includes: For each data currently remaining in the source data set, determine one by one whether the data is less than the last data currently remaining in the target data set. If it is less than, determine that the data belongs to the difference set of the source data with respect to the target data, and determine that the data meets the preset difference set deletion condition, then delete the data from the source data set; if it is greater than, determine that the data currently does not meet the difference set deletion condition, then retain the data in the source data set.
7. A data comparison device, characterized in that: The device includes: A source data receiving module is configured to obtain at least two source data after sorting; wherein the source data is data from a data source; A target data acquisition module, configured to acquire at least two sorted target data; wherein the target data is the data stored in the updated party; The data set processing module is configured to set a source data set and a target data set; whenever the source data set is empty, at least two source data are sequentially collected from all the sorted source data according to a preset fixed data step length, and the source data set in the memory is updated with the ordered source data collected this time; whenever the target data set is empty, at least two target data are sequentially collected from all the sorted target data according to a preset fixed data step length, and the target data set in the memory is updated with the ordered target data collected this time; The comparison processing module is configured to execute in sequence whenever at least one of the source data set and the target data set is updated: first, for each data in the source data set, determine one by one whether the data is the same as a data in the target data set. If so, determine that the data belongs to the intersection of the source data set and the target data set, and delete the data from the source data set and the target data set; then, for each data currently remaining in the source data set, determine one by one whether the data meets the preset difference set deletion condition. If so, delete the data from the source data set; if not, retain the data in the source data set; finally, determine whether the source data set and the target data set are currently empty. If both are not empty, determine that each data currently remaining in the target data set belongs to the difference set of the target data set with respect to the source data set.
8. The device according to claim 7, characterized in that The device further comprises: a synchronization processing module; The sync processing module is configured to perform: After the comparison processing module determines that a piece of data belongs to the intersection of the source data set and the target data set, the synchronization processing module modifies the value of the data in the target data to the value of the data in the source data. and / or, After the comparison processing module determines that a data in the source data set belongs to a difference set of the source data set with respect to the target data set, the synchronization processing module adds the data to the target data set; and / or, After the comparison processing module determines that each data currently remaining in the target data set belongs to a difference set of the target data set with respect to the source data set, the synchronization processing module deletes each data currently remaining in the target data set from the target data.
9. The device according to claim 7, characterized in that When the comparison processing module determines whether the data satisfies the preset difference set deletion condition one by one, it is configured to execute: For each data currently remaining in the source data set, determine one by one whether the data is less than the last data currently remaining in the target data set. If it is less, determine that the data belongs to the difference set of the source data for the target data, and determine that the data meets the preset difference set deletion condition. If it is greater, determine that the data currently does not meet the difference set deletion condition.
10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Data comparing method and device
CN104239301A
Data cross analysis method and device, computer equipment and storage medium
CN112434048A
Data processing method and distributed processing system
CN114297311A
Method and system for synchronization of databases
US20090287726A1
Merging of distributed datasets
US20170083515A1
