Data comparison method and related equipment
By setting up the data set in the database with the same association relationship and traversing the data along the preset direction to obtain the comparison data, the problem of inefficient data comparison in the prior art is solved, and efficient data comparison is achieved.
Patent Information
- Application Number
- CN202410932725.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-15
AI Technical Summary
After the data synchronization or migration between different types of databases, the existing data comparison method needs to be compared and consistently compared with the data set, which is inefficient, especially when the data set is large, all data needs to be traversed to obtain. Comparing the data leads to excessive time consumption.
By setting the data set with the same association relationship in a specific area, using the association relationship to determine the comparison data in the data set, and traversing the comparison data along the preset direction to reduce invalid searches and improve comparison efficiency.
It realizes the acquisition of required comparison data without traversing the entire data set, which significantly improves data comparison efficiency, reduces the number of CPU processing times, and improves comparison efficiency.
Smart Images

Figure CN118964418A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing, and in particular to a data comparison method and related equipment. Background Art
[0002] In the prior art, data synchronization or data migration is required between different types of databases. After the data synchronization or data migration, the stored data of different data sets need to be compared for consistency.
[0003] Each consistency comparison requires obtaining multiple sets of comparison data with corresponding relationships in the two data sets. In the traditional data comparison scheme, multiple sets of comparison data are obtained serially, and each set of comparison data is obtained from each comparison data set according to the corresponding relationship. In order to avoid missing each item of data in the data set, it is necessary to traverse each item of data in the data set each time the comparison data is obtained. When there is a large amount of data in the data set, the traditional data comparison scheme takes a lot of time to compare the data, and the comparison efficiency is low. Summary of the invention
[0004] The embodiments of the present application provide a data comparison method and related devices for improving data comparison efficiency.
[0005] A first aspect of an embodiment of the present application provides a data comparison method, comprising:
[0006] Acquire a first data set and a second data set, wherein the first data set includes a plurality of first data, the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship;
[0007] Centrally setting a plurality of second data having the same association relationship with the first data in the first data set in the same area of the second data set;
[0008] Based on the association relationship, determining multiple groups of comparison data in the first data set respectively;
[0009] For each group of comparison data in the first data set, traverse the second data set according to the corresponding association relationship until all second data in the target comparison area having the same association relationship with each group of comparison data are obtained;
[0010] The first data in each group of comparison data is compared with all the second data in the corresponding target comparison area to obtain a comparison result.
[0011] In a specific implementation, it also includes:
[0012] centrally setting a plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set;
[0013] The determining of a plurality of groups of comparison data in the first data set based on the association relationship comprises:
[0014] Based on the association relationship, a plurality of first comparison regions in the first data set are respectively determined, and the first data in each of the first comparison regions constitutes a group of comparison data.
[0015] In a specific implementation, the determining, based on the association relationship, a plurality of first comparison regions in the first data set includes:
[0016] If a write instruction of the first buffer data set is received, the first data set is traversed along a preset direction, and a set of target comparison data in the unmatched first comparison area is written into the first buffer data set;
[0017] The traversing the second data set according to the corresponding association relationship until all the second data in the target comparison area having the same association relationship with each group of comparison data is obtained includes:
[0018] If the target comparison data is written into the first buffer data set, the second data set is traversed along the preset direction, and all second data in the target comparison area having the same association relationship with the target comparison data is written into the second buffer data set;
[0019] The step of comparing the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result includes:
[0020] Performing data comparison based on the first buffer data set and the second buffer data set, and obtaining corresponding comparison results;
[0021] After obtaining the corresponding comparison result, the first buffer data set and the second buffer data set are cleared, and a new write instruction for the first buffer data set is initiated until there is no uncompared first comparison area in the first data set.
[0022] In a specific implementation, traversing the first data set along a preset direction and writing a group of target comparison data in the unmatched first comparison area into the first buffer data set includes:
[0023] Taking the target starting position in the first comparison data set as the starting point, traversing the first data set along a preset direction, and writing a group of target comparison data in the unmatched first comparison area into the first buffer data set, wherein the first target starting position in the first comparison data set is before the first first data of the first comparison data set along the preset direction;
[0024] If the target comparison data is written into the first buffer data set, the last first data in the target comparison data is determined according to the preset direction, and the target starting position is updated to after the last first data in the target comparison data along the preset direction.
[0025] In a specific implementation, before obtaining the first original data set and the second original data set, the method further includes:
[0026] If a comparison instruction for the first form and the second form is obtained, a preset comparison logic between the first form and the second form is obtained, wherein the comparison logic includes associated fields and associated conditions;
[0027] Determine the identification code of each first entry row or column based on the field value of each first entry row or column under the associated field in the first form, and determine the identification code of each second entry row or column based on the field value of each second entry row or column under the associated field in the second form;
[0028] Determine the identification code of each first entry row or column as the identification code of the first data of the corresponding entry row or column, and determine the identification code of each second entry row or column as the identification code of the second data of the corresponding entry row or column;
[0029] According to the association condition, the association relationship between the identification code of the first data and the identification code of the second data is determined, and according to the association relationship between the identification code of the first data and the identification code of the second data, the association relationship between the first data and the second data is determined.
[0030] In a specific implementation, the step of centrally arranging the plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set includes:
[0031] performing preset processing on the first data set according to the association relationship, the preset processing comprising grouping a plurality of first data in the first original data set that have the same association relationship with the second data, or sorting a plurality of first data in the first original data set that have different association relationships with the second data according to corresponding identification codes;
[0032] The step of centrally arranging a plurality of second data having the same association relationship with the first data in the first data set in the same area of the second data set includes:
[0033] In the second original data set, second data having the same association relationship with the first data are grouped accordingly, or multiple second data having different association relationships with the first data in the second original data set are sorted according to corresponding identification codes.
[0034] In a specific implementation, after obtaining the comparison result of each group of comparison data, the method further includes:
[0035] Associatively storing the first data in each group of comparison data, the second data in the corresponding target comparison area, and the comparison result;
[0036] Data associated with different comparison results are displayed separately, wherein the data associated with the comparison results include first data in the comparison data and second data in the corresponding target comparison area.
[0037] A second aspect of an embodiment of the present application provides a computer device, including:
[0038] An acquisition unit, configured to acquire a first data set and a second data set, wherein the first data set includes a plurality of first data, the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship;
[0039] a processing unit, configured to centrally set a plurality of second data having the same association relationship with the first data in the first data set in a same area of the second data set;
[0040] a determining unit, configured to determine, based on the association relationship, a plurality of groups of comparison data in the first data set, respectively, wherein the first data in each group of the comparison data has the same association relationship with the second data in the second data set;
[0041] a traversal unit, configured to traverse the second data set according to the corresponding association relationship for each group of comparison data in the first data set, until all second data in the target comparison area having the same association relationship with each group of comparison data is acquired;
[0042] The comparison unit is used to compare the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result.
[0043] In a specific implementation, the processing unit is further used to centrally set a plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set;
[0044] The determining unit is specifically configured to determine, based on the association relationship, a plurality of first comparison regions in the first data set, respectively, wherein the first data in each of the first comparison regions is a group of comparison data.
[0045] In a specific implementation, the determining unit is specifically configured to, if receiving a write instruction of the first buffer data set, traverse the first data set along a preset direction, and write a set of target comparison data in the unmatched first comparison area into the first buffer data set;
[0046] The traversal unit is specifically configured to, if the target comparison data is written into the first buffer data set, traverse the second data set along the preset direction, and write all second data in the target comparison area having the same association relationship with the target comparison data into the second buffer data set;
[0047] The comparison unit is specifically used to perform data comparison based on the first buffer data set and the second buffer data set, and obtain a corresponding comparison result;
[0048] After obtaining the corresponding comparison result, the first buffer data set and the second buffer data set are cleared, and a new write instruction for the first buffer data set is initiated until there is no uncompared first comparison area in the first data set.
[0049] In a specific implementation, the determination unit is specifically configured to take the target starting position in the first comparison data set as the starting point, traverse the first data set along a preset direction, and write a group of target comparison data in the unmatched first comparison area into the first buffer data set, wherein the first target starting position in the first comparison data set is before the first first data of the first comparison data set along the preset direction;
[0050] If the target comparison data is written into the first buffer data set, the last first data in the target comparison data is determined according to the preset direction, and the target starting position is updated to after the last first data in the target comparison data along the preset direction.
[0051] In a specific implementation, before acquiring the first original data set and the second original data set, the acquisition unit is further configured to acquire preset comparison logic between the first form and the second form if a comparison instruction for the first form and the second form is obtained, wherein the comparison logic includes associated fields and associated conditions;
[0052] The determining unit is further configured to determine the identification code of each first entry row or column based on the field value of each first entry row or column under the associated field in the first form, and to determine the identification code of each second entry row or column based on the field value of each second entry row or column under the associated field in the second form;
[0053] The determining unit is further used to determine the identification code of each first entry row or column as the identification code of the first data of the corresponding entry row or column, and to determine the identification code of each second entry row or column as the identification code of the second data of the corresponding entry row or column;
[0054] The determination unit is further used to determine the association relationship between the identification code of the first data and the identification code of the second data based on the association condition, and to determine the association relationship between the first data and the second data based on the association relationship between the identification code of the first data and the identification code of the second data.
[0055] In a specific implementation, the processing unit is specifically used to perform preset processing on the first data set according to the association relationship, and the preset processing includes grouping a plurality of first data in the first original data set that have the same association relationship with the second data, or sorting a plurality of first data in the first original data set that have different association relationships with the second data according to corresponding identification codes;
[0056] The processing unit is specifically used to perform grouping processing on the second data set that has the same association relationship with the first data, or to sort multiple second data in the second original data set that has different association relationships with the first data according to corresponding identification codes.
[0057] In a specific implementation, after obtaining the comparison result of each set of comparison data, the computer device further includes a storage unit and a display unit;
[0058] The storage unit is used to associate and store the first data in each group of comparison data, the second data in the corresponding target comparison area, and the comparison result;
[0059] The display unit is used to distinguish and display data associated with different comparison results, and the data associated with the comparison results include first data in the comparison data and second data in the corresponding target comparison area.
[0060] A third aspect of an embodiment of the present application provides a computer device, including:
[0061] CPU, memory and input / output interface;
[0062] The memory is a short-term storage memory or a persistent storage memory;
[0063] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.
[0064] A fourth aspect of the embodiments of the present application provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method described in the first aspect.
[0065] A fifth aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.
[0066] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: for the first data set including multiple first data, the second data set including multiple second data, the first data in the first data set and the second data in the second data set have a preset association relationship. And multiple second data with the same association relationship as the first data in the first data set are centrally set in the same area of the second data set. It can be understood that the second data in the second data set that need to be merged and compared (or have the same association relationship with the first data) are all in the same area in the data set. In this way, when obtaining the second data required for each comparison, it is not necessary to traverse the second data set to know that all the required second data have been obtained, because all the second data required for each comparison are present in a certain concentrated area in the second data set. Compared with the traditional solution, when the second comparison data required for each comparison is compared, the search method of traversing all the data in the data set is more efficient, and thus has a higher comparison efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 A schematic diagram of a flow chart of the data comparison method disclosed in the embodiment of the present application;
[0068] Figure 2 Another schematic diagram of the process of the data comparison method disclosed in the embodiment of the present application;
[0069] Figure 3 Another schematic diagram of the process of the data comparison method disclosed in the embodiment of the present application;
[0070] Figure 4 Another structural schematic diagram of a computer device disclosed in an embodiment of the present application;
[0071] Figure 5Another structural schematic diagram of the computer device disclosed in the embodiment of the present application. DETAILED DESCRIPTION
[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0073] The embodiments of the present application provide a data comparison method and related devices for improving data comparison efficiency.
[0074] The embodiment of the present application provides the following data comparison method for comparing data in two data sets. The first data set includes multiple first data, the second data set includes multiple second data, and there is a preset association relationship between the first data and the second data to be compared. Since in the actual comparison scenario, the data with the same association relationship in the first data set and the second data set, that is, the data that needs to be centrally compared, in order to reduce the cost of finding data with the same association relationship, the present application creatively proposes: the second data corresponding to the same association relationship with any first data is centrally set in a certain area in the second data set, so that when searching for the second data with the same association relationship with each first data, it is not necessary to traverse the entire second data set to ensure that all the required second data have been obtained.
[0075] by Figure 1 Taking the data set shown in the figure as an example, the obtained data set 1 (also called the main table) is the first data set, and each row of data in the obtained data set 1 is a first data. Similarly, the obtained data set 2 (also called the secondary table) is the first data set, and each row of data in the obtained data set 2 is a second data. The association relationship between the first data and the second data is determined by Figure 1 The relational data set shown in FIG. 1 determines that the value of a primary table associated field and the value of a secondary table associated field contained in each row of data in the relational data set constitute an associated relationship.
[0076] Specifically, it is considered that all first data and all second data with the same association relationship need to be centrally compared. The definition of the same association relationship can be determined according to the needs of data comparison, and is not specifically limited here. If it is necessary to compare the first data and the second data in the corresponding association relationship that contain the same primary table association field or secondary table association field, then it is necessary to determine the two association relationships with the same value of the primary table association field or the same value of the secondary table association field as the same association relationship; if it is only necessary to compare the first data and the second data in the corresponding association relationship that contain the same value of the primary table association field and the same value of the secondary table association field, then only the two association relationships with the same value of the primary table association field and the same value of the secondary table association field are determined as the same association relationship. For example, if it is necessary to determine two association relationships with the same value of the primary table association field or the same value of the secondary table association field as the same association relationship, then Figure 1 In the association relationship set shown in , the association relationship formed by the first row of data and the association relationship formed by the fourth row of data are the same association relationship.
[0077] for Figure 1 For the data set 1 and data set 2 shown, the first data and the second data corresponding to the same association relationship in the association relationship set should be compared together. The data in the data set usually obtained may be arranged in the order of actual generation. Therefore, multiple second data with the same association relationship as the first data are scattered throughout the data set 2. In order to completely find the data with the same association relationship, it is necessary to traverse the entire second data set. However, after the multiple second data with the same association relationship as the first data are concentrated in the same area of data set 2, the multiple second data with the same association relationship as the first data are located in the same area of data set 2, such as the second data with the primary key values of the secondary table being 7 and 9 respectively. In this way, when traversing data set 2 during the data comparison process, all the incoming data with the same association relationship as a certain procurement data can be obtained from only a certain comparison area. For example, to obtain the second data that has the same association relationship with the associated field value A in the main table, you only need to traverse the centralized data set 2 from top to bottom, and when you traverse the second data with the associated value H in the secondary table, you will know that you have obtained all the second data in data set 2 that have the same association relationship with the associated field value A in the main table, without having to traverse to the last data in data set 2, which can effectively improve the data comparison efficiency.
[0078] Based on the above examples, please refer to Figure 2 The present application also provides a data comparison method, comprising the following steps:
[0079] 201. Obtain a first data set and a second data set, where the first data set includes a plurality of first data, and the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship.
[0080] First, a first data set and a second data set that need to be compared are obtained, wherein the first data set includes a plurality of first data, and the second data set includes a plurality of second data.
[0081] It should be noted that the embodiments of the present application require that the two data sets for data comparison need to maintain data consistency between the two data sets, or the data in one data set needs to rely on the data in the other data set for data verification, which is not limited here. Among them, the preset association relationship between the first data and the second data can be configured through the field of the first data and the field of the second data, such as the first data with certain values under the associated field and the second data with certain values under the associated field have an association relationship. For details, please refer to the aforementioned related embodiments, which will not be repeated here.
[0082] 202. Set a plurality of second data having the same association relationship with the first data in the first data set in a same area of the second data set.
[0083] According to the above-mentioned embodiments, all first data and all second data with the same association relationship need to be centrally compared. Therefore, in order to improve the cost of finding the required second data during the data comparison process, the embodiment of the present application proposes to centrally set multiple second data with the same association relationship as the first data in the first data set in the same area of the second data set, so that all second data with the same association relationship as any first data can be simply obtained from an area of the second data set without traversing the entire second data set each time. Specifically, this step can use any algorithm or technical means that can centrally set multiple second data with the same association relationship in the same area of the second data set, such as a sorting algorithm.
[0084] For example, Figure 1 Taking the data set 2 after centralized setting as an example, the value of the secondary table associated field is used as the sorting object. Through the sorting algorithm, two second data with the same association relationship, namely, the secondary table primary key 7 and the secondary table primary key 9, can be set in adjacent positions in the data set 2.
[0085] 203. Based on the association relationship, determine multiple groups of comparison data in the first data set respectively.
[0086] According to the contents of the aforementioned embodiments, the first data and the second data having the same association relationship need to be centrally compared. Therefore, this step requires taking the first data having the same association relationship with any second data as a group of comparison data, and processing each first data in the first data set in a certain manner until multiple groups of comparison data are obtained.
[0087] by Figure 1 Taking the data set 1 shown in as an example, there are three groups of comparison data that can be obtained therefrom according to the method of the embodiment of the present application, namely: 1. The first data whose primary key values of the main table are 1, 5, and 6 respectively; 2. The first data whose primary key value of the main table is 3; 3. The first data whose primary key value of the main table is 4.
[0088] 204. For each group of comparison data in the first data set, traverse the second data set according to the corresponding association relationship until all second data in the target comparison area having the same association relationship with each group of comparison data is obtained.
[0089] According to the contents of the aforementioned embodiments, the first data and the second data having the same association relationship need to be centrally compared. Therefore, after obtaining multiple groups of comparison data in step 203, this step also needs to obtain, for each group of comparison data, the second data that needs to be centrally compared with the first data in the group of comparison data.
[0090] by Figure 1 Taking the data set 2 shown in as an example, all the second data in the target comparison area having the same association relationship with each group of comparison data obtained according to the method of the embodiment of the present application are as follows: 1. The second data having the same association relationship with the first data whose primary key values of the main table are 1, 5, and 6 are the second data in the target comparison area 1; 2. The second data having the same association relationship with the first data whose primary key value of the main table is 4 is each second data in the target comparison area 2. Among them, there is no second data in the second data set that has the same association relationship with the first data whose primary key value of the main table is 4. For the processing that can be performed on these comparison data, please refer to the relevant embodiments described below.
[0091] 205. Compare the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result.
[0092] It can be understood that all the first data in a set of comparison data and all the second data in the target comparison area corresponding to the first data in the set of comparison data are all the data in the first data set and the second data set that have a certain same association relationship, and these data are all the data of a certain type that needs to be centrally compared in the first data set and the second data set. Therefore, based on the aforementioned steps 203 and 204, this step can complete the final data comparison, wherein the specific data comparison method is not limited and can be selected and configured as needed.
[0093] It should be noted that if Figure 1 The first data with the primary key value of the main table determined as 3 is the set of comparison data, and Figure 2 The second data whose primary key values of the secondary table are 1, 3, and 2 respectively cannot be compared with any second data or first data. In other words, for such second data that does not have the same association relationship with any first data, or first data that does not have the same association relationship with any second data, it is impossible and unnecessary to compare data. Therefore, the processing of such data can be carried out according to the needs or in accordance with the processing method in the corresponding data comparison method, which is not limited here.
[0094] In the embodiment of the present application, for the first data set including a plurality of first data, the second data set including a plurality of second data, the first data in the first data set and the second data in the second data set have a preset association relationship. And the plurality of second data having the same association relationship as the first data in the first data set are centrally arranged in the same area of the second data set. It is understandable that the second data that need to be merged and compared (or have the same association relationship with the first data) in the second data set are all in the same area in the data set. In this way, when obtaining the second data required for each comparison, it is not necessary to traverse the second data set to know that all the required second data have been obtained, because all the second data required for each comparison are present in a certain concentrated area in the second data set. Compared with the traditional scheme, when comparing the required second comparison data each time, the search method of traversing all the data in the data set, the embodiment of the present application does not need to frequently traverse the data set, thereby streamlining the data comparison process, reducing the number of CPU processing times, and thus having a higher comparison efficiency.
[0095] It is understandable that in addition to the centralized setting method of the second data set, in order to further improve the data query efficiency and thus improve the data comparison efficiency. Based on the aforementioned embodiment, the embodiment of the present application can also perform centralized setting processing on the first data set, and the specific steps are as follows: multiple first data having the same association relationship with the second data in the second data set are centralized in the same area of the first data set; and then, the operation of determining each group of comparison data in step 203 is as follows: based on the association relationship, multiple first comparison areas in the first data set are determined respectively, and the first data in each first comparison area is a group of comparison data.
[0096] Similar to the aforementioned embodiment, after multiple first data having the same association relationship with the second data in the second data set are centrally set in the same area of the first data set, all first data having the same association relationship with any second data can be simply obtained from one area of the first data set without traversing the entire first data set each time. Specifically, this step can use any algorithm or technical means that can centrally set multiple first data having the same association relationship in the same area of the first data set, such as a sorting algorithm. Among them, the basis for the centralized setting of the first data set and the second data set can be the same or different, or the objects for the centralized setting of the first data set and the second data set can be the same or different. Among them, the objects for centralized setting can include but are not limited to sorting objects, grouping objects, etc., which depends on the algorithm selected for the centralized setting and is not limited here.
[0097] For example, Figure 1 Take the data set 1 after centralized setting as an example, select sorting method 1 to use the value of the main table association field as the sorting object (different from the sorting object of the second data set). Through the sorting algorithm, the three first data with the same association relationship between the primary key 1, 5, and 6 of the main table and the same second data can be set in the adjacent position in data set 1; select Figure 1 The sorting method 2 used in the centralized setting is to use the values of the secondary table associated fields as the sorting object (the same as the sorting object of the second data set). Considering that there are no secondary table associated fields in the first data, but there is an association relationship between the values of the primary table associated fields and the values of the secondary table associated fields, or in other words, there is an association between the values of the primary table associated fields and the values of the secondary table associated fields. Therefore, based on this feature, the first data is sorted according to the values of the secondary table associated fields associated with each value of the primary table associated fields. The specific sorting effect is as follows: Figure 1 Similarly, the centralized setting of the second data set of the present application can also take the value of the main table associated field as the object, and sort the second data according to the value of the main table associated field associated with each secondary table associated field value.
[0098] In the prior art, after determining multiple sets of comparison data and target comparison areas with the same association relationship with each set of comparison data, the multiple sets of comparison data and the second data in the target comparison areas with the same association relationship with each set of comparison data are set together in the same comparison data set, and then data comparison is performed. In a business scenario where there is a large amount of first data and a large amount of second data, constructing the aforementioned comparison data set will occupy a large amount of memory space, affecting the overall data comparison efficiency.
[0099] Furthermore, in order to reduce the above-mentioned memory space occupancy problem, the aforementioned step 203 of the embodiment of the present application may include: if a write instruction for the first buffer data set is received, then the first data set is traversed along a preset direction, and a group of target comparison data in the unmatched first comparison area is written into the first buffer data set; further, the aforementioned step 204 may include: if the target comparison data is written into the first buffer data set, then the second data set is traversed along a preset direction, and all second data in the target comparison area having the same association relationship with the target comparison data is written into the second buffer data set; further, the aforementioned step 205 may include: performing data comparison based on the first buffer data set and the second buffer data set, and obtaining corresponding comparison results; after obtaining the corresponding comparison results, clearing the first buffer data set and the second buffer data set, and initiating a new write instruction for the first buffer data set until there is no unmatched first comparison area in the first data set.
[0100] Specifically, the first data set is traversed along a specific direction, and after a set of target comparison data in a first comparison area that has not been compared is obtained, it is written into the first buffer data set. Subsequently, all the second data in the target comparison area that has the same association relationship with the aforementioned target comparison data are found in the second data set and written into the second buffer data set. Then, data comparison is performed based on the first data in the first buffer data set and the second data in the second buffer data set, and corresponding comparison results are obtained. After obtaining the comparison results, the first buffer data set and the second buffer data set can be cleared, and a new write instruction for the first buffer data set can be initiated to obtain a new set of target comparison data, which is compared with the second data in the corresponding target comparison area until there is no unmatched comparison data, or there is no unmatched first comparison area. In the embodiment of the present application, in the same time period, at most all the first data and second data with a certain same association relationship exist in the first buffer data set and the second buffer data set, which can support the comparison of the first data set and the second data set with a very large amount of data without causing server downtime.
[0101] It should be noted that if any group of comparison data has no association with any second data, the first buffer data set can be directly cleared after writing the first buffer data set, and a new write instruction for the first buffer data set can be initiated; or the first comparison area where the group of comparison data is located can be determined as a compared area, and a new write instruction for the first buffer data set can be initiated.
[0102] Furthermore, if the objects centrally set in the first data set and the second data set are the same, there is no need to repeat the traversal when traversing to obtain each unmatched first comparison area and traversing each target comparison area. The following takes "traversing the first data set along a preset direction and writing a set of target comparison data in the unmatched first comparison area into the first buffer data set" as an example to illustrate the technical means for implementing traversal in the embodiment of the present application, including: starting from the target starting position in the first comparison data set, traversing the first data set along a preset direction, writing a set of target comparison data in the unmatched first comparison area into the first buffer data set, the first target starting position in the first comparison data set being before the first first data in the first comparison data set along the preset direction; if the target comparison data is written into the first buffer data set, determining the last first data in the target comparison data according to the preset direction, and updating the target starting position to after the last first data in the target comparison data along the preset direction.
[0103] When traversing the first data set for the first time, the target starting position is taken as the area before the first first data in the first data set along the preset direction, and the first data set is traversed along the preset direction. After a set of target comparison data is continuously written into the first buffer data set (the first comparison area where the target comparison data is located is synchronously updated to the compared area), the area before the first first data in the first first comparison area that has not been compared in the preset direction is determined as the new target starting position.
[0104] by Figure 3 As an example, the preset direction is the direction indicated by the arrow in the figure. Figure 3 When the data set 1 is shown in , the target starting position is before the first data with the primary key of the main table being 1, and the first data with the primary keys of the main table being 1, 5, and 6 in data set 1 are traversed downward in sequence, and the first data with the primary keys of the main table being 1, 5, and 6 are written into the first buffer data set of the buffer as a group of target comparison data, and the result is as follows: Figure 3 Synchronously, after the target comparison data is written into the first buffer data set, the first data with the primary key 3 of the main table needs to be determined as the new target starting position.
[0105] Similarly, the method of writing the second data into the second data set is similar to the aforementioned implementation, except that, in the step of "if the target comparison data is written into the first buffer data set, the second data set is traversed along the preset direction, and all the second data in the target comparison area having the same association relationship with the target comparison data is written into the second buffer data set", when the second data set is traversed for the first time, the target starting position is taken as the area before the first target comparison area in the second data set, and the second data set is traversed along the preset direction until the target comparison area where the second data having the same association relationship with the aforementioned target comparison data is located is found, and all the second data in the target comparison area is written into the second buffer data set, and the position after the aforementioned target comparison area and between the aforementioned target comparison areas in the preset direction are determined as the new target starting position.
[0106] by Figure 3 As an example, after writing the first data with the primary key of the primary table being 1, 5, and 6 into the first buffer data set, according to the association relationship between the value of the associated field of the primary table and the value of the associated field of the secondary table (i.e. Figure 1 ), the target comparison area where the second data whose associated field value is A in the secondary table is located should be traversed, and the second data therein should be obtained and written into the second buffer data set, that is, the second data whose primary key is 8 in the secondary table should be written into the second buffer data set, and the result should be as follows: Figure 3 Synchronously, after the second data in the target comparison area is written into the second buffer data set, the position between the second data with the secondary table primary keys of 8 and 1 needs to be determined as the new target starting position.
[0107] In summary, whether it is the target starting position in the first data set or the target starting position in the second data set, its role is to ensure that each data in the data set is traversed only once. Therefore, the update and setting of the target starting position can be any position that meets the above requirements.
[0108] In some specific implementations, the association relationship between the first data and the second data in the present application can be determined in the following manner: if a comparison instruction for the first form and the second form is obtained, the preset comparison logic between the first form and the second form is obtained, and the comparison logic includes an associated field and an associated condition; based on the field value of each first entry row or column in the first form under the associated field, the identification code of each first entry row or column is determined respectively, and based on the field value of each second entry row or column in the second form under the associated field, the identification code of each second entry row or column is determined respectively; the identification code of each first entry row or column is determined as the identification code of the first data of the corresponding entry row or column, and the identification code of each second entry row or column is determined as the identification code of the second data of the corresponding entry row or column; according to the associated condition, the association relationship between the identification code of the first data and the identification code of the second data is determined, and according to the association relationship between the identification code of the first data and the identification code of the second data, the association relationship between the first data and the second data is determined.
[0109] Specifically, there is a preset comparison logic in the first form and the second form of the two sheets that need to be compared for data, so as to guide the computer to compare the first data with the second data therein. Therefore, after receiving the comparison instructions for the first form and the second form, the preset comparison logic between the two should be obtained first. Generally, the comparison logic includes associated fields and associated conditions. There is a corresponding relationship between the associated fields and associated conditions in each comparison logic. The associated fields include at least one field in the first form and at least one field in the second form, wherein the field of the first form is used to determine the identification code of the first data and the field of the second form is used to determine the identification code of the second data; and the associated conditions are used to assist the first data and the second data in determining the association relationship according to the corresponding identification code. In addition, in order to realize data comparison, an associated field in a comparison logic requires at least one field in the first form and one field in the second form.
[0110] It should be noted that, since the identification code of the data is determined based on the value of the data under the associated field, the specific value of the first entry row or column under the associated field is the value of the fields of the first form included in the first entry row or column in the associated field, and the value of the second entry row or column under the associated field is the value of the fields of the second form included in the second entry row or column in the associated field. For example, if the associated field includes field 1 of the first form and field 2 of the second form, then the identification code of the first entry row or column is the value of field 1 in the first entry row or column, and the identification code of the second entry row or column is the value of field 2 in the second entry row or column. Furthermore, the identification code of the first entry row or column may be the value of the first entry row or column under the associated field itself, or a preset identification code corresponding to the value of the first entry row or column under the associated field, etc., which is any form that can make the first entry row or column with different values under the associated field have different identification codes; similarly, the identification code of the second entry row or column may be the value of the second entry row or column under the associated field itself, or a preset identification code corresponding to the value of the second entry row or column under the associated field, etc., which is any form that can make the second entry row or column with different values under the associated field have different identification codes, and is not limited here.
[0111] In addition, a form usually includes entry rows or columns, and an entry row or column usually represents a piece of data in the form. Therefore, the identification code of each first entry row or column in the first form can be used as the identification code of the first data corresponding to the first entry row or column. Similarly, the identification code of each second entry row or column in the second form can be used as the identification code of the second data corresponding to the second entry row or column.
[0112] Finally, the association condition is used to indicate the conditions that need to be met between two identification codes that have an association relationship, wherein one of the two identification codes is the identification code of the first data and the other is the identification code of the second data. In this way, for the first data and the second data whose corresponding identification codes satisfy the association relationship, there is an association relationship between the identification codes corresponding to the two (the first data and the second data), and further, it can be said that there is an association relationship between the two.
[0113] On the basis of the above-mentioned embodiments, in some specific implementations, the centralized setting of the data with the same association relationship in the first data set and the second data set can be realized by any means such as grouping or sorting, as described below: the first data set is preset according to the association relationship, and the preset processing includes grouping the multiple first data with the same association relationship with the second data in the first original data set, or sorting the multiple first data with different association relationships with the second data in the first original data set according to the corresponding identification code; in the second original data set, the second data with the same association relationship with the first data is correspondingly grouped, or the multiple second data with different association relationships with the first data in the second original data set is sorted according to the corresponding identification code. Among them, the first data in the first original data set is the same as the data in the first data set, and the second data in the second original data set is the same as the data in the second data set.
[0114] Generally, the first original data set and the second original data set can be directly obtained from a server or a database. However, as described in the aforementioned embodiments, the data sets directly obtained from the server or the database, wherein the data having the same association relationship is scattered throughout the data sets, therefore, in order to reduce the data comparison cost, the present application can adopt methods including but not limited to sorting processing and grouping processing for both the first data set and the second data set, and centrally arrange the data in the data sets based on the association relationship. For details, please refer to the aforementioned related embodiments, which will not be repeated here.
[0115] In actual applications, after the aforementioned step 205, the embodiment of the present application also includes: associating and storing the first data in each group of comparison data, the second data in the corresponding target comparison area, and the comparison result; distinguishing and displaying the data associated with different comparison results, the data associated with the comparison result includes the first data in the comparison data and the second data in the corresponding target comparison area.
[0116] Specifically, the comparison results and the data associated with the comparison results are displayed visually. At the same time, the associated data is displayed differently according to the type of the comparison result. For example, if there are two types of comparison results, one is a completed comparison and the other is an incomplete comparison, then the data associated with the comparison result of the completed comparison type is displayed in yellow, and the data associated with the comparison result of the incomplete comparison type is displayed in red. Figure 4 Specifically, in addition to using different colors to distinguish and display, the data associated with different types of comparison results displayed can also be distinguished by fonts, etc., which is not specifically limited here. In addition, the result set of data associated with different comparison results displayed in the embodiment of the present application can be stored in a data set with a compression algorithm to reduce the storage space occupied.
[0117] See also Figure 4 According to an embodiment of the present application, a computer device is provided, including:
[0118] An acquisition unit 401 is used to acquire a first data set and a second data set, wherein the first data set includes a plurality of first data, the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship;
[0119] The processing unit 402 is used to centrally set a plurality of second data having the same association relationship with the first data in the first data set in the same area of the second data set;
[0120] A determination unit 403 is used to determine, based on the association relationship, multiple groups of comparison data in the first data set, wherein the first data in each group of comparison data has the same association relationship with the second data in the second data set;
[0121] A traversal unit 404 is used to traverse the second data set according to the corresponding association relationship for each group of comparison data in the first data set, until all second data in the target comparison area having the same association relationship with each group of comparison data is obtained;
[0122] The comparison unit 405 is used to compare the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result.
[0123] In a specific implementation, the processing unit 402 is further configured to centrally set a plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set;
[0124] The determination unit 403 is specifically configured to determine, based on the association relationship, a plurality of first comparison regions in the first data set, respectively, wherein the first data in each first comparison region is a group of comparison data.
[0125] In a specific implementation, the determination unit 403 is specifically configured to, if receiving a write instruction of the first buffer data set, traverse the first data set along a preset direction, and write a set of target comparison data in the unmatched first comparison area into the first buffer data set;
[0126] The traversal unit 404 is specifically configured to traverse the second data set along a preset direction if the target comparison data is written into the first buffer data set, and write all second data in the target comparison area having the same association relationship with the target comparison data into the second buffer data set;
[0127] The comparison unit 405 is specifically used to perform data comparison based on the first buffer data set and the second buffer data set, and obtain corresponding comparison results;
[0128] After obtaining the corresponding comparison result, the first buffer data set and the second buffer data set are cleared, and a new write instruction for the first buffer data set is initiated until there is no uncompared first comparison area in the first data set.
[0129] In a specific implementation, the determination unit 403 is specifically configured to take the target starting position in the first comparison data set as the starting point, traverse the first data set along a preset direction, and write a group of target comparison data in the unmatched first comparison area into the first buffer data set, wherein the first target starting position in the first comparison data set is before the first first data of the first comparison data set along the preset direction;
[0130] If the target comparison data is written into the first buffer data set, the last first data in the target comparison data is determined according to a preset direction, and the target starting position is updated to after the last first data in the target comparison data along the preset direction.
[0131] In a specific implementation, before obtaining the first original data set and the second original data set, the obtaining unit 401 is further configured to obtain a preset comparison logic between the first form and the second form if a comparison instruction for the first form and the second form is obtained, wherein the comparison logic includes an associated field and an associated condition;
[0132] The determining unit 403 is further used to determine the identification code of each first entry row or column based on the field value of each first entry row or column under the associated field in the first form, and to determine the identification code of each second entry row or column based on the field value of each second entry row or column under the associated field in the second form;
[0133] The determining unit 403 is further used to determine the identification code of each first entry row or column as the identification code of the first data of the corresponding entry row or column, and determine the identification code of each second entry row or column as the identification code of the second data of the corresponding entry row or column;
[0134] The determination unit 403 is further used to determine the association relationship between the identification code of the first data and the identification code of the second data according to the association condition, and determine the association relationship between the first data and the second data according to the association relationship between the identification code of the first data and the identification code of the second data.
[0135] In a specific implementation, the processing unit 402 is specifically configured to perform a preset process on the first data set according to the association relationship, wherein the preset process includes grouping a plurality of first data having the same association relationship with the second data in the first original data set, or sorting a plurality of first data having different association relationships with the second data in the first original data set according to corresponding identification codes;
[0136] The processing unit 402 is specifically used to perform grouping processing on the second data having the same association relationship with the first data in the second original data set, or to sort multiple second data having different association relationships with the first data in the second original data set according to corresponding identification codes.
[0137] In a specific implementation, after obtaining the comparison result of each set of comparison data, the computer device further includes a storage unit and a display unit;
[0138] A storage unit, used for performing associated storage on the first data in each group of comparison data, the second data in the corresponding target comparison area, and the comparison result;
[0139] The display unit is used to distinguish and display data associated with different comparison results, where the data associated with the comparison results include first data in the comparison data and second data in the corresponding target comparison area.
[0140] Figure 5 It is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 500 may include one or more central processing units (CPU) 501 and a memory 505. The memory 505 stores one or more application programs or data.
[0141] The memory 505 may be a volatile storage or a persistent storage. The program stored in the memory 505 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit 501 may be configured to communicate with the memory 505 and execute a series of instruction operations in the memory 505 on the computer device 500.
[0142] The computer device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0143] The CPU 601 can execute the aforementioned Figures 1 to 4 The operations performed by the computer device in the illustrated embodiment will not be described in detail here.
[0144] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the instructions of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0146] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
[0150] An embodiment of the present application also provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer executes the data comparison method as described above.
Claims
1. A data comparison method, characterized in that: include: Acquire a first data set and a second data set, wherein the first data set includes a plurality of first data, the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship; Centrally setting a plurality of second data having the same association relationship with the first data in the first data set in the same area of the second data set; Based on the association relationship, determining multiple groups of comparison data in the first data set respectively; For each group of comparison data in the first data set, traverse the second data set according to the corresponding association relationship until all second data in the target comparison area having the same association relationship with each group of comparison data are obtained; The first data in each group of comparison data is compared with all the second data in the corresponding target comparison area to obtain a comparison result.
2. The data comparison method according to claim 1, characterized in that: Also includes: centrally setting a plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set; The determining of a plurality of groups of comparison data in the first data set based on the association relationship comprises: Based on the association relationship, a plurality of first comparison regions in the first data set are respectively determined, and the first data in each of the first comparison regions constitutes a group of comparison data.
3. The data comparison method according to claim 2, characterized in that: The determining, based on the association relationship, a plurality of first comparison regions in the first data set respectively includes: If a write instruction of the first buffer data set is received, the first data set is traversed along a preset direction, and a set of target comparison data in the unmatched first comparison area is written into the first buffer data set; The traversing the second data set according to the corresponding association relationship until all the second data in the target comparison area having the same association relationship with each group of comparison data is obtained includes: If the target comparison data is written into the first buffer data set, the second data set is traversed along the preset direction, and all second data in the target comparison area having the same association relationship with the target comparison data is written into the second buffer data set; The step of comparing the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result includes: Performing data comparison based on the first buffer data set and the second buffer data set, and obtaining corresponding comparison results; After obtaining the corresponding comparison result, the first buffer data set and the second buffer data set are cleared, and a new write instruction for the first buffer data set is initiated until there is no uncompared first comparison area in the first data set.
4. The data comparison method according to claim 3, characterized in that: The traversing the first data set along a preset direction and writing a set of target comparison data in the unmatched first comparison area into the first buffer data set includes: Taking the target starting position in the first comparison data set as the starting point, traversing the first data set along a preset direction, and writing a group of target comparison data in the unmatched first comparison area into the first buffer data set, wherein the first target starting position in the first comparison data set is before the first first data of the first comparison data set along the preset direction; If the target comparison data is written into the first buffer data set, the last first data in the target comparison data is determined according to the preset direction, and the target starting position is updated to after the last first data in the target comparison data along the preset direction.
5. The data comparison method according to claim 4, characterized in that: Before obtaining the first original data set and the second original data set, the method further includes: If a comparison instruction for the first form and the second form is obtained, a preset comparison logic between the first form and the second form is obtained, wherein the comparison logic includes associated fields and associated conditions; Determine the identification code of each first entry row or column based on the field value of each first entry row or column under the associated field in the first form, and determine the identification code of each second entry row or column based on the field value of each second entry row or column under the associated field in the second form; Determine the identification code of each first entry row or column as the identification code of the first data of the corresponding entry row or column, and determine the identification code of each second entry row or column as the identification code of the second data of the corresponding entry row or column; According to the association condition, the association relationship between the identification code of the first data and the identification code of the second data is determined, and according to the association relationship between the identification code of the first data and the identification code of the second data, the association relationship between the first data and the second data is determined.
6. The data comparison method according to claim 5, characterized in that: The step of centrally arranging the plurality of first data having the same association relationship with the second data in the second data set in the same area of the first data set includes: performing preset processing on the first data set according to the association relationship, the preset processing comprising grouping a plurality of first data in the first original data set that have the same association relationship with the second data, or sorting a plurality of first data in the first original data set that have different association relationships with the second data according to corresponding identification codes; The step of centrally arranging a plurality of second data having the same association relationship with the first data in the first data set in the same area of the second data set includes: In the second original data set, second data having the same association relationship with the first data are grouped accordingly, or multiple second data having different association relationships with the first data in the second original data set are sorted according to corresponding identification codes.
7. The data comparison method according to any one of claims 1 to 6, characterized in that: After obtaining the comparison result of each group of comparison data, the method further includes: Associatively storing the first data in each group of comparison data, the second data in the corresponding target comparison area, and the comparison result; Data associated with different comparison results are displayed separately, wherein the data associated with the comparison results include first data in the comparison data and second data in the corresponding target comparison area.
8. A computer device, characterized in that: include: An acquisition unit, configured to acquire a first data set and a second data set, wherein the first data set includes a plurality of first data, the second data set includes a plurality of second data, and the first data in the first data set and the second data in the second data set have a preset association relationship; a processing unit, configured to centrally set a plurality of second data having the same association relationship with the first data in the first data set in a same area of the second data set; a determining unit, configured to determine, based on the association relationship, a plurality of groups of comparison data in the first data set, respectively, wherein the first data in each group of the comparison data has the same association relationship with the second data in the second data set; a traversal unit, configured to traverse the second data set according to the corresponding association relationship for each group of comparison data in the first data set, until all second data in the target comparison area having the same association relationship with each group of comparison data is acquired; The comparison unit is used to compare the first data in each group of comparison data with all the second data in the corresponding target comparison area to obtain a comparison result.
9. A computer device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the data comparison method according to any one of claims 1 to 7.
10. A computer program product comprising instructions, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the data comparison method according to any one of claims 1 to 7.
11. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the data comparison method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Data verification method and system
CN115422180A
Data comparison method and device, medium and electronic equipment
CN115510062A
Data association analysis method and device, electronic equipment and computer storage medium
CN116701438A
Ranking datasets based on data attributes
US20220197914A1
