Database consistency comparison and compensation method, device, equipment, medium and product

Abstract

The invention discloses a database consistency comparison and compensation method. The method comprises the following steps: acquiring a plurality of databases to be verified; sorting each piece of data in each to-be-verified database according to the primary key, and calculating a hash value of each piece of data in each sorted database to obtain a plurality of preprocessed databases; comparing the plurality of preprocessed databases by adopting a comparison algorithm based on the primary key and the hash value to determine a comparison result; the comparison result is a first result or a second result; the first result represents that the plurality of preprocessed databases have no difference; the second result represents that the plurality of preprocessed databases have differences; the second result comprises a difference conclusion and comparison information; and generating a difference file based on the comparison information, and compensating the plurality of preprocessed databases based on the difference file to obtain a plurality of compensated databases. The detection efficiency is improved, and the storage cost and the resource overhead are reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database consistency comparison and compensation method, device, equipment, medium and product. Background Art

[0002] As the scale of system business continues to grow, business stability has attracted much attention. Most companies choose to deploy multiple sets of database services to improve the availability and disaster recovery capabilities of the system, so that when a computer room fails, the business can quickly switch to another computer room to work normally. Therefore, data consistency verification between databases in each computer room is particularly important. However, with the increase in transaction volume year by year, data inconsistency between multiple databases is inevitable, and data differences are difficult to be discovered in time, making data comparison and verification more and more difficult. In this case, if the database switching operation is performed, it will affect the normal operation of the business system, and in some cases, it may also cause serious production accidents.

[0003] In order to solve the database consistency problem, the data copy method is generally used, using multiple data copies and multiple historical data copies for comparison to obtain multiple modification record sets, and then performing data comparison and compensation based on multiple modification record sets. However, this method has high resource overhead, consumes more storage space, and has low efficiency, making it unsuitable for practical use. Summary of the invention

[0004] Based on the above problems, the present application provides a database consistency comparison and compensation method, device, equipment, medium and product, the purpose of which is to improve detection efficiency and reduce storage costs and resource overhead.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] The first aspect of the present application provides a database consistency comparison and compensation method, the method comprising:

[0007] Obtain multiple databases to be verified;

[0008] Sort each piece of data in each database to be verified according to the primary key and calculate the hash value of each piece of data in each sorted database to obtain multiple preprocessed databases;

[0009] A comparison algorithm is used to compare multiple preprocessed databases based on primary keys and hash values ​​to determine a comparison result; the comparison result is a first result or a second result; the first result indicates that there is no difference between the multiple preprocessed databases; the second result indicates that there are differences between the multiple preprocessed databases; the second result includes a conclusion of the difference and comparison information;

[0010] A difference file is generated based on the comparison information, and multiple preprocessed databases are compensated based on the difference file to obtain multiple compensated databases.

[0011] Optionally, the using a comparison algorithm to compare multiple pre-processed databases based on primary keys and hash values ​​to determine a comparison result specifically includes:

[0012] Using any one of the multiple pre-processed databases as a standard database and using the remaining databases as databases to be compared;

[0013] Each piece of data in the standard database is compared with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result.

[0014] Optionally, the step of comparing each piece of data in the standard database with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result specifically includes:

[0015] Determine whether the primary key and hash value of each piece of data in each database to be compared meet a first preset condition, and obtain a first judgment result; the first preset condition is that the primary keys and hash values ​​of the data in the database to be compared and the data corresponding to the standard database are consistent;

[0016] If the first judgment result is yes, it indicates that the plurality of data to be compared are consistent with the standard database, and the comparison result is the first result;

[0017] If the first judgment result is no, the characterization comparison result is a second result, and the comparison information in the second result is generated by comparing each data in the standard database with each data in the database to be compared based on the comparison rule based on the primary key and the hash value.

[0018] Optionally, the step of comparing each piece of data in the standard database with each piece of data in the database to be compared based on the comparison rule based on the primary key and the hash value to generate the comparison information in the second result specifically includes:

[0019] Determine whether the data in the database to be compared that does not meet the first preset condition in the multiple databases to be compared meet the second preset condition, and obtain a second judgment result; the second preset condition is that the primary keys of the data in the database to be compared and the data corresponding to the standard database are consistent and the hash values ​​are inconsistent;

[0020] If the second judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the second preset condition is recorded as inconsistent;

[0021] If the second judgment result is no, determine whether the data in the database to be compared that does not meet the first preset condition and the second preset condition in the multiple databases to be compared meet the third preset condition, and obtain a third judgment result; the third preset condition is that the primary key of the data in the database to be compared is smaller than the primary key of the data corresponding to the standard database or the total number of data items in the data to be verified is greater than the total number of data items in the standard data;

[0022] If the third judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the third preset condition is recorded as redundant;

[0023] If the third judgment result is no, the comparison information of the corresponding data in the to-be-compared database that does not meet the third preset condition is recorded as missing.

[0024] Optionally, compensating a plurality of preprocessed databases based on the difference file to obtain a plurality of compensated databases specifically includes:

[0025] Parse the difference file to obtain the parsed file;

[0026] The compensation method for each piece of data in each database to be compared is determined based on the parsed file, and each database to be compared is compensated based on a preset compensation method to obtain a plurality of compensated databases.

[0027] Optionally, determining a compensation method for each piece of data in each database to be compared based on the parsed file, and compensating each database to be compared based on a preset compensation method to obtain multiple compensated databases specifically includes:

[0028] Based on the parsed file, determine whether each piece of data in each to-be-compared database has inconsistency, and obtain a fourth determination result;

[0029] If the fourth judgment result is yes, updating the inconsistent data based on the standard database;

[0030] If the fourth judgment result is no, then judging whether each piece of data in each database to be compared has redundancy, and obtaining a fifth judgment result;

[0031] If the fifth judgment result is yes, deleting redundant data;

[0032] If the fifth judgment result is no, indicating that there are missing data in the multiple databases to be compared, the missing data are added based on the standard database.

[0033] A second aspect of the present application provides a database consistency comparison and compensation device, the device comprising:

[0034] An acquisition unit, used for acquiring multiple databases to be verified;

[0035] A preprocessing unit, used to sort each piece of data in each database to be verified according to the primary key and calculate the hash value of each piece of data in each sorted database to obtain multiple preprocessed databases;

[0036] A comparison unit, used for comparing the multiple pre-processed databases based on the primary key and the hash value using a comparison algorithm to determine a comparison result; the comparison result is a first result or a second result; the first result indicates that there is no difference between the multiple pre-processed databases; the second result indicates that there are differences between the multiple pre-processed databases; the second result includes a conclusion of the existence of differences and comparison information;

[0037] The compensation unit is used to generate a difference file based on the comparison information, and compensate the multiple preprocessed databases based on the difference file to obtain multiple compensated databases.

[0038] A third aspect of the present application provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned database consistency comparison and compensation method.

[0039] A fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned database consistency comparison and compensation method is implemented.

[0040] A fifth aspect of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned database consistency comparison and compensation method is implemented.

[0041] Compared with the prior art, this application has the following beneficial effects:

[0042] The present application sorts the data of each database according to the primary key, ensuring that the basis for comparison between databases is consistent during subsequent comparison and compensation, and realizes difference comparison of data in different environments and different database types, and then calculates the hash value of each piece of data in the database to be verified, and determines the comparison result by gradually comparing the primary key and hash value of each piece of data. The comparison result is the first result or the second result. If the comparison result is the first result, it indicates that there is no difference between the databases to be verified. If the comparison result is the second result, it indicates that there is a difference between the databases to be verified, and the second result can also reflect the comparison information, which is convenient for compensation between databases. Therefore, the present application does not require multiple modification record sets and data copies during the comparison and compensation process, and has high detection efficiency, low storage cost and low resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A flowchart of a database consistency comparison and compensation method provided in an embodiment of the present application;

[0045] Figure 2 A first schematic diagram of a comparison algorithm of a database consistency comparison and compensation method provided in an embodiment of the present application;

[0046] Figure 3 A second schematic diagram of a comparison algorithm of the database consistency comparison and compensation method provided in an embodiment of the present application;

[0047] Figure 4 A structural diagram of a database consistency comparison and compensation device provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of the preprocessing unit and the comparison unit provided in the embodiment of the present application;

[0049] Figure 6 A flowchart of the compensation unit provided in the embodiment of the present application;

[0050] Figure 7 A flowchart of the control unit provided in the embodiment of the present application;

[0051] Figure 8 A flowchart of the abnormal monitoring unit provided in the embodiment of the present application;

[0052] Fig. 9 This is a workflow diagram of the database consistency comparison and compensation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] As described above, the current multi-database comparison and compensation resource overhead is large, consumes a lot of storage space, and has low efficiency, which is not suitable for practical use. After research, the inventor found that the database to be verified is sorted according to the primary key, and the hash value of each data in the database to be verified is calculated separately. By gradually comparing the primary key and hash value of each data to determine the comparison result, high detection efficiency, low storage cost and low resource overhead can be achieved.

[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Explanation of relevant terms:

[0056] Data compensation: When some data in the database is lost or erroneous, restore the lost or erroneous data to ensure the accuracy and integrity of the data.

[0057] Disaster recovery: Strategies and measures to ensure that enterprise businesses and applications can be quickly and effectively restored and continue normal operations when they are hit by natural disasters, man-made damage, cyber attacks, and other emergencies.

[0058] Data backup: The process of copying data in a database to other storage media.

[0059] Data source: refers to the place or source where data is stored and provided.

[0060] Time complexity: used to describe the running time of an algorithm when processing data of different sizes.

[0061] Space complexity: A measure used to describe the additional storage space required by an algorithm during its operation.

[0062] At present, some known technical solutions for database consistency comparison and compensation are generally:

[0063] Select the target database from the data source module and collect data, perform consistency check on the collected data and generate a differentiation script. Although this method supports comparing the structural differences between multiple databases and supports one-click generation of differentiation scripts, it directly compares the source data. When the data scale is large, the system overhead is also large. In addition, when compensating data, the differentiation script generated by comparison automatically compensates the data, which makes it difficult to ensure the accuracy of the differentiation script.

[0064] Alternatively, configure the data source for data comparison, compare the two tables of source and destination data, calculate the total number of rows, and then perform hash calculations on the two tables of source and destination data, extract the data of each row in the table, and generate a unique hash value for each row of data. If the hash value of the destination does not exist in the source table, mark this row of data as inconsistent, and generate a difference script for manual data compensation. Although this method simplifies data comparison, it does not support difference comparison of data in different environments and different database types, and lacks preprocessing of source data. The comparison effect is poor for data tables with large data volumes, and manual compensation for differences requires additional manpower.

[0065] Alternatively, the first data copy is compared with the corresponding first historical data copy to obtain a first modification record set; the second data copy is compared with the corresponding second historical data copy to obtain a second modification record set; and the data comparison result between the first data copy and the second data copy is obtained based on the historical comparison result, the first modification record set, and the second modification record set. Although this method overcomes the data difference, it uses a data copy method and needs to maintain multiple files. When the data is compared again later, a copy file needs to be generated again, and the source and copy files need to be compared and analyzed. The space complexity is high, the system overhead is large, and the waste of resources and space consumption are greatly increased.

[0066] In view of the shortcomings of the above-mentioned prior art, the inventors have conducted relevant research and proposed a database consistency comparison and compensation method, which overcomes the technical problems existing in the prior art. Figure 1 The flowchart of the database consistency comparison and compensation method shown in FIG. 1 illustrates a database consistency comparison and compensation method:

[0067] S101. Acquire multiple databases to be verified.

[0068] In actual applications, multiple computer room databases of the same enterprise can be selected as databases to be verified, so as to achieve comparison and compensation of multiple computer room databases of the same enterprise. When a failure occurs in one of the computer rooms of the enterprise, by applying the database consistency comparison and compensation method provided by the technical solution of the present application, the business can be quickly switched to another computer room database for normal operation without affecting the normal operation of the business.

[0069] S102 , sorting each piece of data in each database to be verified according to the primary key and calculating the hash value of each piece of data in each sorted database to obtain a plurality of preprocessed databases.

[0070] First determine the primary key of each data in each database to be verified, and then sort each data in the database to be verified according to the primary key, ensuring that the basis for comparison between databases is consistent during subsequent comparison and compensation, and realizing difference comparison of data in different environments and different database types.

[0071] The hash value of each data in the sorted database is calculated in order to compare the hash value of each data during the comparison process. There is no need to compare each data in each data, so as to avoid occupying too many resources during the comparison process. In addition, the present application does not need to generate a copy for comparison during the comparison process, which reduces the system overhead and effectively saves operating resources and storage space.

[0072] S103: Using a comparison algorithm to compare multiple pre-processed databases based on primary keys and hash values ​​to determine a comparison result.

[0073] This application divides the comparison results into two types, namely the first result or the second result. If the comparison result is the first result, it indicates that there is no difference between the multiple preprocessed databases, and there is no need to compensate between the databases, so there is no need to generate a difference file for compensation. If the comparison result is the second result, it indicates that there are differences between the multiple preprocessed databases, and it is necessary to use a comparison rule to compare the multiple preprocessed databases to determine the difference file, and then perform data compensation on the multiple preprocessed databases based on the difference file. Therefore, the second result needs to include a conclusion that there is a difference and comparison information. This difference conclusion is a conclusion that there is a difference, and this comparison information can be understood as the comparison information in the difference file, and the difference file is constructed based on the comparison information.

[0074] In order to overcome the problems of the prior art, the comparison method of this application is improved. Now the comparison algorithm of this application is described:

[0075] Any one of the multiple preprocessed databases is used as a standard database, and the remaining databases are used as databases to be compared.

[0076] The comparison algorithm of the present application adopts a one-party-based comparison method, which means comparing and compensating other databases based on any one of multiple databases. Therefore, the present application needs to select a standard database as the basis for comparison and compensation, and the selection of the standard database of the present application mainly adopts two methods: random selection and customization.

[0077] If the time interval between the last comparison and compensation of multiple preprocessed databases is short or the probability of differences between multiple preprocessed databases is small, the standard database can be determined by random selection.

[0078] If the interval between the last comparison and compensation of multiple preprocessed databases is long or there is a high probability of differences between multiple preprocessed databases, any database can be customized as a standard database, for example, the currently running database can be used as the standard database.

[0079] Each piece of data in the standard database is compared with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result.

[0080] The comparison result is generated by comparing the primary key of each data in the standard database with the primary key of each database in each database to be compared, and by comparing the hash value of each data in the standard database with the hash value of each database in each database to be compared.

[0081] It is determined whether the primary key and hash value of each piece of data in each database to be compared meet a first preset condition to obtain a first determination result. The first preset condition is that the primary key and hash value of the data in the database to be compared are consistent with the data corresponding to the standard database.

[0082] If the first judgment result is yes, it indicates that the plurality of data to be compared are consistent with the standard database, and the comparison result is the first result.

[0083] If the first preset condition is met, it indicates that the data in each database to be compared is consistent with the data in the standard database, and there is no need to generate a difference file, and the comparison result is the first result.

[0084] If the first judgment result is no, the characterization comparison result is a second result, and the comparison information in the second result is generated by comparing each data in the standard database with each data in the database to be compared based on the comparison rule based on the primary key and the hash value.

[0085] The comparison rules of this application overcome the problem of excessive resource occupation in the prior art. The comparison rules of this application are now explained:

[0086] Determine whether the data in the database to be compared that does not meet the first preset condition among multiple databases to be compared meets the second preset condition, and obtain a second judgment result; the second preset condition is that the primary keys of the data in the database to be compared and the data corresponding to the standard database are consistent, but the hash values ​​are inconsistent.

[0087] If the second judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the second preset condition is recorded as inconsistent.

[0088] If the second judgment result is no, determine whether the data in the database to be compared that does not meet the first preset condition and the second preset condition among the multiple databases to be compared meet the third preset condition to obtain a third judgment result; the third preset condition is that the primary key of the data in the database to be compared is smaller than the primary key of the data corresponding to the standard database or the total number of data items in the data to be verified is greater than the total number of data items in the standard data.

[0089] If the third judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the third preset condition is recorded as redundant.

[0090] If the third judgment result is no, the comparison information of the corresponding data in the to-be-compared database that does not meet the third preset condition is recorded as missing.

[0091] The comparison rule of the present application is mainly to compare whether the primary key and hash value of each data in the standard database and the database to be compared are consistent.

[0092] If the primary key of the data in the standard database is consistent with the primary key of the database to be compared but the hash value is inconsistent, it means that there is a data inconsistency problem in the data to be compared. Therefore, it is necessary to record the inconsistency in the database to be compared in the comparison information.

[0093] If the primary key of the database to be compared is greater than the primary key of the data in the standard database or the total number of data in the database to be compared is greater than the total number of data in the standard database, it means that there is data redundancy in the database to be compared, and the existence of redundancy in the database to be compared needs to be recorded in the comparison information.

[0094] If the primary key of the database to be compared is smaller than the primary key of the data in the standard database or the total number of data entries in the database to be compared is smaller than the total number of data entries in the standard database, it means that there is a problem of missing data in the database to be compared, and the missing data in the database to be compared needs to be recorded in the comparison information.

[0095] This application provides a specific implementation of the comparison algorithm, using a pointer to point to the data to be compared and the standard database for comparison, and more intuitively presents the comparison algorithm. Figure 2 The first schematic diagram of the alignment algorithm of the consistency alignment and compensation method shown and Figure 3 The second schematic diagram of the comparison algorithm of the database consistency comparison and compensation method shown in the figure explains the comparison algorithm:

[0096] Under the premise that the primary key is in order, such as Figure 2 As shown, the standard database is defined as Figure 2 The database 1 shown in the figure defines the database to be compared as Figure 2 The database 2 to database n shown in the figure, where key is the primary key and value is the hash value. The comparison algorithm flow is as follows Figure 3 As shown:

[0097] (1) Pointer p1 points to primary key key11 in the primary key list of database 1, pointer p2 points to primary key key21 in the primary key list of database 2, and pointer pn points to primary key keyn1 in the primary key list of database n. Based on database 1, compare the primary key and the hash value corresponding to the primary key table by table to see if they are the same.

[0098] (2) Based on the primary key key11 pointed to by pointer p1, if the primary key keyi1 corresponding to a table pointer pi is smaller than the primary key key11 of pointer p1, it means that the data does not exist in database 1. The database needs to perform a data deletion operation, the comparison information record is redundant, and the pointer pi is moved downward.

[0099] (3) If the pointer pi points to the primary key keyi1 and the pointer p1 points to the primary key key11, and the hash value valuei1 corresponding to the primary key pointed to by the pointer pi is consistent with the hash value value11 corresponding to the primary key pointed to by the pointer p1, then the data is consistent. Compare the primary keys pointed to by other database pointers in turn.

[0100] (4) If the pointer pi points to the primary key keyi1 and the pointer p1 points to the primary key key11, but the hash value valuei1 corresponding to the primary key pointed to by the pointer pi is inconsistent with the hash value value11 corresponding to the primary key pointed to by the pointer p1, then based on database 1, the database needs to perform a data update operation, and the comparison result is recorded as inconsistent. The primary keys pointed to by the pointers of other data tables are compared in turn.

[0101] (5) If the pointer pi points to the primary key keyi1, which is greater than the pointer p1 points to the primary key key11, it means that the data is missing in the database. The data table needs to be added with new data. The comparison result is recorded as missing. The primary keys pointed to by the pointers of other data tables are compared in turn.

[0102] (6) If the last database has been compared in this round, the pointer p1 moves downward and (2)-(5) are repeated.

[0103] (7) If pointer p1 has traversed all the data tables 1 and pointer pi has traversed all the data tables i, then it is necessary to delete the remaining uncompared data from the i+1th to the nth data in the database, and the comparison information record is redundant.

[0104] (8) If pointer p1 has traversed all the data table 1 and all other database table pointers have been traversed, the comparison process ends.

[0105] S104 , generating a difference file based on the comparison information, and compensating the plurality of preprocessed databases based on the difference file to obtain a plurality of compensated databases.

[0106] The specific compensation process of this application is as follows:

[0107] Parse the difference file to obtain the parsed file.

[0108] The compensation method for each piece of data in each database to be compared is determined based on the parsed file, and each database to be compared is compensated based on a preset compensation method to obtain a plurality of compensated databases.

[0109] Compensation methods include adding, updating and deleting.

[0110] Based on the parsed file, it is determined whether there is inconsistency in each piece of data in each database to be compared, and a fourth determination result is obtained.

[0111] If the fourth judgment result is yes, the inconsistent data is updated based on the standard database.

[0112] If there is data in the database to be compared that is inconsistent with the standard database, the inconsistent database needs to be updated based on the standard database, that is, the inconsistent data is deleted and the corresponding data in the standard database is copied and pasted to the deleted part.

[0113] If the fourth judgment result is no, then it is determined whether each piece of data in each database to be compared has redundancy, to obtain a fifth judgment result.

[0114] If the fifth judgment result is yes, the redundant data is deleted.

[0115] Deletion indicates that the database to be compared contains data that is not present in the standard database in addition to the content that is consistent with the standard database, and this data needs to be deleted.

[0116] If the fifth judgment result is no, indicating that there are missing data in the multiple databases to be compared, the missing data are added based on the standard database.

[0117] A new feature is to supplement the missing data database based on the standard database, that is, to find the data corresponding to the missing part in the missing data database in the standard database, and copy and paste this part of the data into the missing data database.

[0118] In specific applications, the database consistency comparison and compensation method may also include:

[0119] Verify user permissions. When receiving a configuration request for user data consistency comparison, verify user application permissions.

[0120] Database comparison and compensation can only be performed when the user has application permissions, which can ensure the security of the database.

[0121] Set the execution time of the data consistency comparison process, and conduct regular comparisons to ensure that multiple databases continue to maintain data consistency, providing reliable data support for the normal operation of comparison and compensation.

[0122] Flexibly select the database and data entities for comparison in the data source.

[0123] When an exception is detected, an alarm is generated and an exception log is recorded. The exception type is determined and analyzed based on the exception log. The automatic compensation process is closed and the compensation is ended, which is convenient for the gateway system operation and maintenance support personnel to troubleshoot the problem. When no exception is detected, the process ends.

[0124] This application supports comparing data differences between multiple databases, and performs consistency comparison on data by comparing hash values. It has high detection efficiency, low storage cost, low memory usage, and no need to directly compare source data. When the data scale is large, the system overhead is also small. It supports difference comparisons in different environments and different database types, increases preprocessing operations on source data, reduces the cost of data comparison, and can automatically compensate data according to the generated difference files, reducing manual operations and increasing the flexibility of compensation operations. There is no need to store data copies, and there is no need to repeat comparison and calculation operations on multiple tables, which reduces the waste of system resources and space consumption, has low storage costs, and occupies less system overhead.

[0125] The present application also provides a database consistency comparison and compensation device, see Figure 4 The structure diagram of the database consistency comparison and compensation device shown in FIG. 1 is used to explain the database consistency comparison and compensation device:

[0126] The acquisition unit is used to acquire multiple databases to be verified.

[0127] The preprocessing unit is used to sort each piece of data in each database to be verified according to the primary key and calculate the hash value of each piece of data in each sorted database to obtain multiple preprocessed databases.

[0128] The preprocessing unit includes a data source processor and a sorter.

[0129] The data source processor is used to pre-process the input data to solve the problem of different processing and representation methods of the same data caused by differences in data models, data storage, data sorting, etc. between different databases, thereby improving data quality.

[0130] The sorter is used to preprocess the preprocessed database according to primary key sorting and hash calculation.

[0131] The comparison unit is used to compare a plurality of pre-processed databases based on primary keys and hash values ​​using a comparison algorithm to determine a comparison result.

[0132] The comparison result is a first result or a second result; the first result indicates that there is no difference between the multiple preprocessed databases; the second result indicates that there are differences between the multiple preprocessed databases; the second result includes a conclusion of the difference and comparison information.

[0133] The comparison unit includes a difference file generator and a comparer.

[0134] The comparator is used to execute the data comparison process, based on one party's database, and perform data consistency verification according to the comparison algorithm.

[0135] The difference file generator is used to record the difference data between multiple libraries and generate a difference file when the comparison result is the second result.

[0136] The comparer includes a classification module and a comparison module. The functions of the classification module and the comparison module are described below:

[0137] The classification module is used to use any one of the multiple pre-processed databases as a standard database and the remaining databases as databases to be compared.

[0138] The comparison module is used to compare each piece of data in the standard database with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result.

[0139] The comparison module includes a first level judgment module and a second level judgment module. The functions of the first level judgment module and the second level judgment module are now described:

[0140] The first level judgment module is used to judge whether the primary key and hash value of each data in each database to be compared meet the first preset condition to obtain a first judgment result. The first preset condition is that the primary key and hash value of the data in the database to be compared are consistent with the data corresponding to the standard database. If the first judgment result is yes, it indicates that the multiple data to be compared are consistent with the standard database, and the comparison result is the first result.

[0141] The second first-level judgment module is used to characterize the comparison result as a second result if the first judgment result is no, and to generate comparison information in the second result by comparing each data in the standard database with each data in the database to be compared based on the comparison rule based on the primary key and the hash value.

[0142] The second first-level judgment module also includes a first second-level judgment module and a second second-level judgment module. The functions of the first second-level judgment module and the second second-level judgment module are now described:

[0143] The first judgment secondary module is used to judge whether the data in the database to be compared that does not meet the first preset condition among multiple databases to be compared meets the second preset condition, and obtain a second judgment result; if the second judgment result is yes, the comparison information of the corresponding data in the database to be compared that meets the second preset condition is recorded as inconsistent.

[0144] The second preset condition is that the primary keys of the data in the database to be compared and the data corresponding to the standard database are consistent but the hash values ​​are inconsistent.

[0145] The second judgment secondary module is used to judge whether the data in the database to be compared that does not meet the first preset condition and the second preset condition in the multiple databases to be compared meet the third preset condition if the second judgment result is no, and obtain a third judgment result; if the third judgment result is yes, the comparison information of the corresponding data in the database to be compared that meets the third preset condition is recorded as redundant. If the third judgment result is no, the comparison information of the corresponding data in the database to be compared that does not meet the third preset condition is recorded as missing.

[0146] The third preset condition is that the primary key of the data in the database to be compared is smaller than the primary key of the data corresponding to the standard database or the total number of data items in the data to be verified is greater than the total number of data items in the standard data.

[0147] The workflow of the preprocessing unit and the comparison unit is as follows: Figure 5 As shown, the data source processor preprocesses the selected comparison database, the sorter sorts the preprocessed database by primary key, performs hash calculation on the sorted database, compares the database, and generates a comparison result.

[0148] The compensation unit is used to generate a difference file based on the comparison information, and compensate the multiple preprocessed databases based on the difference file to obtain multiple compensated databases.

[0149] The compensation unit includes a difference file parser and a data compensator.

[0150] The compensation unit is used to perform data compensation operations after comparing the data inconsistencies between databases. First, the difference file is parsed, and corresponding operations are performed according to the different difference types recorded in the difference list to ensure the correctness of data compensation.

[0151] The working process of the compensation unit is as follows: Figure 6 As shown, the generated difference file is parsed and automatic data compensation is performed to check whether an abnormality occurs during the compensation process. If so, a problem list is generated and an abnormality analysis is performed. Otherwise, the process ends.

[0152] The difference file parser is used to parse the difference file and obtain the parsed file. The operation types of data compensation can be divided into three categories of "addition, deletion and modification" according to the comparison process.

[0153] The data compensator is used to automatically complete the compensation process according to the difference file. Based on the parsed file, the compensation method of each data in each database to be compared is determined, and each database to be compared is compensated based on the preset compensation method to obtain multiple compensated databases.

[0154] The data compensator includes a first compensation module, a second compensation module and a third compensation module. The functions of the first compensation module, the second compensation module and the third compensation module are now described:

[0155] The first compensation module is used to determine whether each piece of data in each database to be compared has inconsistency based on the parsed file, and obtain a fourth determination result. If the fourth determination result is yes, the inconsistent data is updated based on the standard database.

[0156] The second compensation module is used to determine whether each piece of data in each to-be-compared database is redundant if the fourth determination result is no, and obtain a fifth determination result. If the fifth determination result is yes, delete the redundant data.

[0157] The third compensation module is used to add the missing data based on the standard database if the fifth judgment result is no, indicating that there are missing data in the multiple databases to be compared.

[0158] Compensation errors may occur during the operation of the data compensator. Therefore, in specific applications, the database consistency comparison and compensation device can also be equipped with a control unit and an abnormality monitoring unit according to actual conditions. The entire device is controlled by the control unit. The compensation process is monitored by the abnormality monitoring unit, and when an abnormal situation is found, an abnormal alarm message is sent to suspend the data compensation process, perform abnormal analysis, and restart the data compensation process after the abnormality is resolved.

[0159] See now Figure 4 The structure diagram of the database consistency comparison and compensation device shown in the figure explains the control unit and the abnormality monitoring unit:

[0160] The controller includes a permission checker, an entity selector, a timer, and a compensation starter and stopper.

[0161] The control unit is mainly used to coordinate and control the entire database consistency comparison and compensation device. It configures the timer start time through the user of the authority verifier, selects the database and entity type for data consistency comparison, receives the abnormal alarm information generated when an abnormality occurs during the automatic data compensation process, and the compensation starter and stopper interrupts the automatic compensation process.

[0162] The permission controller is used to verify user permissions. When receiving a user data consistency comparison configuration request, the permission verifier verifies the user's application permissions.

[0163] The timer is used to control the execution time of the data consistency comparison process. Regular comparison can ensure that databases in multiple locations continue to maintain data consistency and provide reliable data support for the normal operation of database consistency comparison and compensation devices.

[0164] The entity selector is used to flexibly select the database and data entities for comparison in the data source.

[0165] The compensation starter and stopper is mainly used to start / stop the automatic data compensation process. When an abnormal alarm message is received from the abnormal monitoring unit during the compensation process, the compensation starter and stopper stops data compensation to prevent data inconsistency between multiple databases due to unsuccessful data synchronization during the data synchronization process.

[0166] The working flow diagram of the control unit is as follows: Figure 7 As shown, user authority control, the entity selector selects the data and entity to be compared, the timer starts the data comparison, an abnormal alarm is generated during the data compensation process, the compensation starter and stopper stops the data compensation, and after the abnormal analysis is completed, the compensation starter and stopper starts the data compensation.

[0167] The abnormal monitoring unit includes an abnormality analyzer, an abnormality alarm and an abnormality listener.

[0168] The abnormality monitoring unit is used to monitor the execution of data compensation in real time. When an abnormality is detected, an abnormality alarm will be generated, and the compensation starter and stopper will interrupt the automatic compensation process. At the same time, the cause of the abnormality will be recorded to generate a problem list to facilitate error troubleshooting.

[0169] The abnormal monitoring unit monitors the abnormal situation during the automatic compensation in real time. The working process of the abnormal monitoring unit is as follows: Figure 8 shown.

[0170] The exception monitoring unit is started synchronously with the control unit to check whether the exception listener detects the exception. When an exception is detected, an alarm is generated and an exception log is recorded, and the exception type is determined based on the exception log. The control unit receives the exception alarm and analyzes the exception log. The compensation starter and stopper turns off the automatic compensation process to end the compensation, which is convenient for the gateway system operation and maintenance support personnel to troubleshoot problems. When no exception is detected, it ends.

[0171] The workflow of the database consistency comparison and compensation device is as follows Fig. 9 As shown:

[0172] The control unit controls the start time of the data comparison timer, and the user who passes the authority verification selects the data source and entity type that need to be compared for consistency.

[0173] The preprocessing unit first performs preprocessing operations on the input data to avoid inconsistent input parameters due to the different storage methods of special values ​​in various database products and inconsistent data sorting in different databases, which is mistakenly judged as difference data. Then, the preprocessed database is sorted according to the primary key and the hash value is calculated to solve the problem of data between different databases being mistakenly judged as difference data due to different data sorting.

[0174] The comparison unit executes the data comparison process, takes one party's database as the standard, performs data consistency verification according to the comparison rules, records the difference data between multiple databases and forms a difference file.

[0175] The compensation unit parses the generated difference files, automatically executes the data compensation process, and realizes data consistency compensation and repair among multiple databases.

[0176] The abnormality monitoring unit monitors the execution of data compensation in real time. When an abnormality is detected, the abnormality alarm generates an abnormal alarm and sends an error message to the control unit. The compensation starter and stopper interrupts the compensation process and generates a problem list. The problem list is analyzed for abnormalities, and the type of abnormality is determined to perform subsequent corresponding operations.

[0177] A computer device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a database consistency comparison and compensation method.

[0178] A computer-readable storage medium stores a computer program, which implements a database consistency comparison and compensation method when executed by a processor.

[0179] A computer program product includes a computer program, and when the computer program is executed by a processor, a database consistency comparison and compensation method is implemented.

[0180] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A database consistency comparison and compensation method, characterized in that: The database consistency comparison and compensation method includes: Obtain multiple databases to be verified; Sort each piece of data in each database to be verified according to the primary key and calculate the hash value of each piece of data in each sorted database to obtain multiple preprocessed databases; A comparison algorithm is used to compare multiple preprocessed databases based on primary keys and hash values ​​to determine a comparison result; the comparison result is a first result or a second result; the first result indicates that there is no difference between the multiple preprocessed databases; the second result indicates that there are differences between the multiple preprocessed databases; the second result includes a conclusion of the difference and comparison information; A difference file is generated based on the comparison information, and multiple preprocessed databases are compensated based on the difference file to obtain multiple compensated databases.

2. The database consistency comparison and compensation method according to claim 1, characterized in that: The comparison algorithm is used to compare multiple pre-processed databases based on primary keys and hash values ​​to determine the comparison results, specifically including: Using any one of the multiple pre-processed databases as a standard database and using the remaining databases as databases to be compared; Each piece of data in the standard database is compared with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result.

3. The database consistency comparison and compensation method according to claim 2, characterized in that: The step of comparing each piece of data in the standard database with each piece of data in the database to be compared based on the primary key and the hash value to generate a comparison result specifically includes: Determine whether the primary key and hash value of each piece of data in each database to be compared meet a first preset condition, and obtain a first judgment result; the first preset condition is that the primary keys and hash values ​​of the data in the database to be compared and the data corresponding to the standard database are consistent; If the first judgment result is yes, it indicates that the plurality of data to be compared are consistent with the standard database, and the comparison result is the first result; If the first judgment result is no, the characterization comparison result is a second result, and the comparison information in the second result is generated by comparing each data in the standard database with each data in the database to be compared based on the comparison rule based on the primary key and the hash value.

4. The database consistency comparison and compensation method according to claim 3, characterized in that: The step of comparing each piece of data in the standard database with each piece of data in the database to be compared based on the comparison rule based on the primary key and the hash value to generate the comparison information in the second result specifically includes: Determine whether the data in the database to be compared that does not meet the first preset condition in the multiple databases to be compared meet the second preset condition, and obtain a second judgment result; the second preset condition is that the primary keys of the data in the database to be compared and the data corresponding to the standard database are consistent and the hash values ​​are inconsistent; If the second judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the second preset condition is recorded as inconsistent; If the second judgment result is no, determine whether the data in the database to be compared that does not meet the first preset condition and the second preset condition in the multiple databases to be compared meet the third preset condition, and obtain a third judgment result; the third preset condition is that the primary key of the data in the database to be compared is smaller than the primary key of the data corresponding to the standard database or the total number of data items in the data to be verified is greater than the total number of data items in the standard data; If the third judgment result is yes, the comparison information of the corresponding data in the to-be-compared database that meets the third preset condition is recorded as redundant; If the third judgment result is no, the comparison information of the corresponding data in the to-be-compared database that does not meet the third preset condition is recorded as missing.

5. The database consistency comparison and compensation method according to claim 4, characterized in that: The compensating a plurality of preprocessed databases based on the difference file to obtain a plurality of compensated databases specifically includes: Parse the difference file to obtain the parsed file; The compensation method for each piece of data in each database to be compared is determined based on the parsed file, and each database to be compared is compensated based on a preset compensation method to obtain a plurality of compensated databases.

6. The database consistency comparison and compensation method according to claim 5, characterized in that: The method of determining the compensation method for each piece of data in each database to be compared based on the parsed file, and compensating each database to be compared based on a preset compensation method to obtain multiple compensated databases specifically includes: Based on the parsed file, determine whether each piece of data in each to-be-compared database has inconsistency, and obtain a fourth determination result; If the fourth judgment result is yes, updating the inconsistent data based on the standard database; If the fourth judgment result is no, then judging whether each piece of data in each database to be compared has redundancy, and obtaining a fifth judgment result; If the fifth judgment result is yes, deleting redundant data; If the fifth judgment result is no, indicating that there are missing data in the multiple databases to be compared, the missing data are added based on the standard database.

7. A database consistency comparison and compensation device, characterized in that: The database consistency comparison and compensation device comprises: An acquisition unit, used for acquiring multiple databases to be verified; A preprocessing unit, used to sort each piece of data in each database to be verified according to the primary key and calculate the hash value of each piece of data in each sorted database to obtain multiple preprocessed databases; A comparison unit, used for comparing the multiple pre-processed databases based on the primary key and the hash value using a comparison algorithm to determine a comparison result; the comparison result is a first result or a second result; the first result indicates that there is no difference between the multiple pre-processed databases; the second result indicates that there are differences between the multiple pre-processed databases; the second result includes a conclusion of the existence of differences and comparison information; The compensation unit is used to generate a difference file based on the comparison information, and compensate the multiple preprocessed databases based on the difference file to obtain multiple compensated databases.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the database consistency comparison and compensation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the database consistency comparison and compensation method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the database consistency comparison and compensation method described in any one of claims 1 to 6 is implemented.

Images