Data consistency checking method, device, equipment and storage medium

Abstract

The application provides a data consistency verification method, device and equipment and a storage medium. The method comprises the following steps: dividing a verification period according to the number of sampling time periods to obtain the sampling time periods, and executing a core query script of the core script execution times in the sampling time periods; according to a hierarchical sampling algorithm, sampling the data of source databases and target databases queried by the core query script at N time points in each sampling time period, and performing data volume consistency verification to obtain a data volume consistency verification result; if the data volume consistency verification result is data volume consistency, performing data content consistency verification on the data of the source databases and the target databases queried by the core query script to obtain a data content consistency verification result. In the method, a specific number of time points are selected in each time period for sampling verification through hierarchical sampling, so as to reduce the database pressure, find data problems in time through periodic verification, improve the effectiveness of verification, and reduce the database pressure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and in particular to a data consistency verification method, device and equipment and a storage medium. BACKGROUND

[0002] With the growth of business, enterprise systems become more and more complex. Complex distributed system architecture is prone to remote call failure, message sending failure and other exceptions, which may cause data inconsistency between systems. In addition, as time goes by, the number of users of the system is growing, and the amount of data of the system is growing. In order to ensure the performance of the core function while retaining historical data, the storage layer gradually evolves into a hot and cold architecture, the hot library stores the core hot data, and the cold library stores the historical data, which also poses a challenge to the data consistency between the hot and cold libraries.

[0003] The redundancy data consistency verification scheme commonly used in the prior art generally adopts an online mode, that is, a field is specially set in a database table to save the checksum of a record, and the checksum is updated at the same time when the record is updated, and the consistency is verified through the checksum field. In addition, there are also offline modes based on business characteristics, which mainly verify the consistency in quantity by checking the number of original data. The redundancy data consistency verification scheme for offline mode is decoupled from the business, but it is low in effectiveness and will bring additional pressure to the database of the storage layer. SUMMARY

[0004] The main purpose of the present application is to solve the technical problems of the existing offline mode redundancy data consistency verification scheme, which is low in effectiveness and will bring additional pressure to the database of the storage layer.

[0005] The first aspect of the present application provides a data consistency verification method, which comprises:

[0006] obtaining a preset data consistency verification parameter and a core query script, wherein the data consistency verification parameter comprises a core script execution frequency, a verification period and a number of sampling time periods, and the core query script is used to query the data of a source database and a target database;

[0007] dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the core script execution frequency in the sampling time period;

[0008] According to a preset hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and data quantity consistency verification is performed to obtain a data quantity consistency verification result, wherein N is a natural number not less than 1.

[0009] If the data volume consistency check result is data volume consistency, data content consistency check is performed on the data of the source database and the target database queried by the core query script to obtain a data content consistency check result.

[0010] The data consistency check result is obtained according to the data content consistency check result.

[0011] The second aspect of the present application provides a data consistency check device, comprising:

[0012] The acquisition module is configured to acquire preset data consistency check parameters and a core query script, wherein the data consistency check parameters include a core script execution number, a check period and a number of sampling time periods, and the core query script is used to query data of a source database and a target database.

[0013] The query module is configured to divide the check period according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the core script execution number in the sampling time period.

[0014] The data volume check module is configured to sample data of the source database and the target database queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and perform data volume consistency check to obtain a data volume consistency check result, wherein N is a natural number not less than 1.

[0015] The data content check module is configured to, when the data volume consistency check result is data volume consistency, perform data content consistency check on the data of the source database and the target database queried by the core query script to obtain a data content consistency check result.

[0016] The check result generation module is configured to obtain a data consistency check result according to the data content consistency check result.

[0017] The third aspect of the present application provides a data consistency check device, comprising a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected by a circuit; the at least one processor calls the instructions in the memory to make the data consistency check device execute the steps of the above-mentioned data consistency check method.

[0018] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores instructions, when it runs on a computer, makes the computer execute the steps of the above-mentioned data consistency check method.

[0019] The data consistency verification method, device, equipment and storage medium, by dividing the verification period according to the number of sampling time periods, obtaining the sampling time period, and executing the core query script of the core script execution times in the sampling time period; according to the hierarchical sampling algorithm, sampling the data of the source database and the target database queried by the core query script at N time points in each sampling time period, and performing data volume consistency verification to obtain the data volume consistency verification result; if the data volume consistency verification result is data volume consistent, the data content consistency verification of the data of the source database and the target database queried by the core query script is performed, and the data content consistency verification result is obtained. In the mode, a specific number of time points are selected in each time period for sampling verification by hierarchical sampling, thereby reducing the database pressure, discovering data problems in time through periodic verification, improving the effectiveness of verification and reducing the database pressure.

[0020] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0021] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The first embodiment of the data consistency verification method in the embodiments of the present application is shown in the figure;

[0023] Figure 2 The first embodiment of the data consistency verification method in the embodiments of the present application is shown in the figure;

[0024] Figure 3 The first embodiment of the data consistency verification method in the embodiments of the present application is shown in the figure;

[0025] Figure 4 The first embodiment of the data consistency verification method in the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] The embodiment of the present application provides a data consistency verification method, device, equipment and storage medium, a verification period is obtained by dividing the verification period according to the number of sampling time periods, and a core query script of the core script execution times is executed in the sampling time period; according to the hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and the data volume consistency verification is performed, and the data volume consistency verification result is obtained; if the data volume consistency verification result is data volume consistency, the data content consistency verification of the data of the source database and the target database queried by the core query script is performed, and the data content consistency verification result is obtained. In the mode, a specific number of time points are selected in each time period for sampling verification by hierarchical sampling, so as to reduce the database pressure, the data problems are found in time by periodic verification, the effectiveness of the verification is improved, and the database pressure is reduced.

[0027] The terms "first", "second", "third", "fourth" and the like in the description, claims, and above drawings of the present application, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so herein is merely for distinguishing between like objects and not necessarily for describing a particular sequential or chronological order. Unless otherwise defined, all terms used herein have the same meaning as they would to one skilled in the art in which the present application is related. It will be understood that any described features can be combined in one or more embodiments.

[0028] For the convenience of understanding, the specific flow of the embodiment of the present application is described below, please refer to Figure 1 The first embodiment of the data consistency verification method in the embodiment of the present application comprises:

[0029] 101, obtain the preset data consistency verification parameter and the core query script, wherein the data consistency verification parameter comprises the core script execution times, the verification period and the number of sampling time periods, and the core query script is used for querying the data of the source database and the target database;

[0030] In the embodiment, the worker configures the data consistency checking task in advance, the data consistency checking task includes the data consistency checking parameters set by the worker and the core query script, when the server or system or data consistency checking device executes the scheme process, the data consistency checking parameters and the core query script are obtained from the worker configured data consistency checking task, wherein the data consistency checking parameters include the source database and target database address, the core script execution times, the checking period, the sampling interval time period number and the retry times, and the like, in addition, other parameters about data consistency checking can also be included, the application is not limited, the core query script is used for querying the data of the source database and the target database,

[0031] In actual application, the source database and the target database are two databases which need to keep data consistency, for example, in order to guarantee the performance of core function and reserve historical data, the storage layer gradually evolves into hot and cold two architectures, the hot library stores core popular data, and the cold library stores historical data, wherein the hot library can be used as the source database, and the cold library can be used as the target database.

[0032] 102. The checking period is divided according to the number of sampling time periods, the sampling time period is obtained, and the core query script is queried according to the core script execution times in the sampling time period;

[0033] In the embodiment, the checking period is the period time segment which needs to be checked for data consistency, for example, the checking period is 1 hour, the data consistency between the source database and the target database needs to be checked for multiple times within 1 hour after the server or system or data consistency checking device executes the scheme process, and the checking period is divided according to the number of sampling time periods, for example, the sampling time period is 5, the checking period is 1 hour, then each 12 minutes in the checking period is a sampling time period, the core query script is executed to query the data of the source database and the target database in each sampling time period, and the execution times are the core script execution times of the data consistency checking parameters set in advance, the data consistency problem between the source database and the target database is found in time through periodic checking.

[0034] 103. According to the preset hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and data volume consistency checking is performed, to obtain a data volume consistency checking result;

[0035] In the embodiment, in the process of data consistency verification, first, data volume consistency verification is performed, because data content consistency verification needs to be encoded, and the resources needed for one-to-one comparison of data content are large, so first, data volume consistency verification is performed, when the data volume is inconsistent, it can be directly determined that the data of the source database and the target database is inconsistent, and in the process of data volume consistency verification, in order to further reduce the database pressure, the data volume of each sampling time period is not directly compared, but through hierarchical sampling, a preset number of time points are taken from the sampling time period, and the data of the time points in the source database and the target database queried by the core query script is obtained for comparison, when the verification result is that the data volume of the two at these time points is consistent, the next step is performed, if not, delay retry is performed.

[0036] In the embodiment, after the data volume consistency verification result is obtained by taking N time points in each sampling time period according to the preset hierarchical sampling algorithm, sampling the data of the source database and the target database queried by the core query script, and performing data volume consistency verification, it further includes: according to the core query script query data volume of the source database and the target database queried by the core script execution times in each sampling time period, filtering t sampling time periods with the largest query data volume from each sampling time period in the verification period, wherein t is a natural number not less than 1; according to the hierarchical sampling algorithm, taking M time points in each of the t sampling time periods with the largest query data volume to sample the data of the source database and the target database queried by the core query script, wherein M is a natural number not less than N; performing data volume consistency verification on the sampling data of the source database and the target database sampled at M time points, and updating the data volume consistency verification result once.

[0037] In the embodiment, if the data volume consistency verification result after one update is that the data volume of the sampling data sampled at M time points is consistent, after the data volume consistency verification result is updated once, it further includes: obtaining the total query time corresponding to the data of the source database and the target database queried by the core query script in each sampling time period; according to the total query time and the core script execution times corresponding to each sampling time period, calculating the average time consumption and the proportional time consumption of each sampling time period.

[0038] Further, after the average time consumption and the proportional time consumption of each sampling time period are calculated according to the total query time and the number of core script executions corresponding to each sampling time period, the method further comprises: selecting t sampling time periods with the highest average time consumption and / or proportional time consumption from the sampling time periods in the verification period; sampling the data of the source database and the target database queried by the core query script at W time points in each of the t sampling time periods according to the hierarchical sampling algorithm, wherein W is a natural number not less than N; performing data volume consistency verification on the sampling data of the source database and the target database at the W time points, and performing secondary updating on the data volume consistency verification result.

[0039] Specifically, in actual application, directly verifying the data of x sampling time periods may bring a large load to the database. For example, if the verification period is 1 hour and the sampling time period is 5, directly performing data volume verification on the 5 time periods means directly comparing the data of 12 minutes, which may be very large and may cause an increase in database pressure, reduce the speed of the entire verification process, and even affect online functions. Therefore, a certain number of time points are selected in each time period for statistics, and when the data volumes of these time points are consistent, more intensive and detailed time points are selected for verification in the time period with high data density and high database load. Because the higher the data density or the greater the load, the lower the stability, the possibility of data inconsistency is greater.

[0040] In the embodiment, the data consistency verification parameter further comprises a retry number; after the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period according to the preset hierarchical sampling algorithm and data volume consistency verification is performed to obtain a data volume consistency verification result, the method further comprises: if the data volume consistency verification result or the data volume consistency verification result after the first updating or the data volume consistency verification result after the second updating is inconsistent, after a preset time delay, the data of the source database and the target database is queried again according to the data consistency verification parameter and the core query script, and data volume consistency verification is performed until the data volume consistency verification result is consistent or the number of times of requerying reaches the retry number.

[0041] Specifically, the mechanism of time delay and retry is adopted in the embodiment to be compatible with scenarios such as a sharded collection. When the data volume consistency verification result or the data volume consistency verification result after the first updating or the data volume consistency verification result after the second updating is inconsistent, the mechanism of time delay and retry is triggered until the number of times of time delay and retry reaches the set retry number.

[0042] After the data quantity consistency check is performed until the data quantity consistency check result is data quantity consistency or the number of times of re-querying reaches the retry number, the method further comprises:

[0043] If the data quantity consistency check result is inconsistency after the number of times of re-querying reaches the retry number, a data consistency check result is obtained according to the data quantity check result.

[0044] Specifically, if the data quantity check result is still inconsistency after repeated query, a data consistency check result is directly generated according to the data quantity check result, and a subsequent data content consistency check process is not needed, thereby saving computing resources.

[0045] 104、If the data quantity consistency check result is data quantity consistency, data content consistency check is performed on data of the source database and the target database queried by the core query script to obtain a data content consistency check result;

[0046] In this embodiment, the data content consistency check performed on the data of the source database and the target database queried by the core query script to obtain a data content consistency check result comprises: y pieces of data queried by the core query script at each sampling time period are sampled from the target database and the source database respectively according to the hierarchical sampling algorithm, wherein y is a natural number not less than 1; and data content consistency check is performed on the y pieces of sampled data in the target database and the source database to obtain a data content consistency check result.

[0047] Further, the data content consistency check performed on the y pieces of sampled data in the target database and the source database to obtain a data content consistency check result comprises: CRC encoding is performed on the y pieces of sampled data in the target database and the source database respectively to obtain corresponding encoding information; and data content consistency check is performed on the corresponding encoding information of the source database and the target database to obtain a data content consistency check result.

[0048] Specifically, in actual application, CRC is a cyclic redundancy check code, which is a commonly used check code in the field of computer network communication. CRC encoding is performed through the cyclic redundancy check code. The cyclic redundancy check code includes a series of data encoding rules such as shifting and division, and the algorithm principle, algorithm program design and analysis can be solved through corresponding software coding. The CRC encoding calculates a check code according to the principle of modulo 2 division, and then the obtained check code is attached to the data to be sent to form a new frame of data, which is sent to the receiving end. The receiving end recalculates the check code according to the rule of the generated check code. If they are the same, it means that the data content is consistent. If they are different, it means that the data content is inconsistent.

[0049] In the embodiment, if the data content consistency check result is consistent encoding information, after the y pieces of sampled data in the target database and the source database are subjected to data content consistency check to obtain a data content consistency check result, the method further comprises: performing data content consistency check on the sampled data of the source database and the target database at M time points, and updating the data content consistency check result once; and if the data content consistency check result after the first update is consistent for the sampled data at the M time points, performing data content consistency check on the sampled data of the source database and the target database at W time points, and updating the data content consistency check result twice.

[0050] 105. obtaining a data consistency check result according to the data content consistency check result.

[0051] In the embodiment, if the data consistency check result is inconsistent, after the data consistency check result is obtained according to the data content consistency check result, the method further comprises: recording the data consistency check result in a preset record log, and executing a preset automatic execution script to obtain a corresponding bottom-up compensation script according to the automatic execution script; obtaining a sampling time period in which the data consistency check is inconsistent according to the bottom-up compensation script, and determining a to-be-compensated data segment of the target database according to the sampling time period in which the data consistency check is inconsistent; and obtaining corresponding data of the to-be-compensated data segment from the source database and copying the data to the target database.

[0052] Specifically, when the data consistency check result is that the data between the source database and the target database is inconsistent in data volume or data content, the data inconsistent with the source database in the target database is compensated by the bottom-up compensation script, and the bottom-up compensation script is an offline data compensation method. According to a bottom-up compensation script address cs provided by a business party, a time range in which data compensation is required is transmitted by remote calling, and the bottom-up compensation script copies the data in this time range to the target database, wherein the time range in which data compensation is required is the sampling time period in which the compared data is inconsistent in the data volume consistency check or the data content consistency check process, the data queried in the source database is determined through the sampling time period, and the data is copied to the target database to complete the bottom-up compensation.

[0053] Specifically, after the data volume consistency check and the data content consistency check are performed, if the obtained data consistency check result is consistent, it indicates that the data in the source database and the target database is consistent, at this time, the resources occupied by the data volume consistency check and the data content consistency check are released, and the program is exited.

[0054] In the embodiment, the sampling time period is obtained by dividing the verification period according to the number of sampling time periods, and the core query script is executed in the sampling time period; according to the hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and the data volume consistency verification is performed to obtain the data volume consistency verification result; if the data volume consistency verification result is data volume consistent, the data content consistency verification is performed on the data of the source database and the target database queried by the core query script to obtain the data content consistency verification result. In the method, the specific number of time points is selected in each time period for sampling verification by hierarchical sampling, thereby reducing the database pressure, discovering data problems in time through periodic verification, improving the effectiveness of verification, and reducing the database pressure.

[0055] The data consistency verification method in the embodiment of the application is described above, and the data consistency verification device in the embodiment of the application is described below, please refer to Figure 2 An embodiment of the data consistency verification device in the embodiment of the application includes:

[0056] The acquisition module 201 is configured to acquire preset data consistency verification parameters and a core query script, wherein the data consistency verification parameters include the number of core script executions, a verification period, and the number of sampling time periods, and the core query script is used to query the data of a source database and a target database;

[0057] The query module 202 is configured to divide the verification period according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the number of core script executions in the sampling time period;

[0058] The data volume verification module 203 is configured to sample the data of the source database and the target database queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and perform data volume consistency verification to obtain a data volume consistency verification result, wherein N is a natural number greater than or equal to 1;

[0059] The data content verification module 204 is configured to, when the data volume consistency verification result is data volume consistent, perform data content consistency verification on the data of the source database and the target database queried by the core query script to obtain a data content consistency verification result;

[0060] The verification result generation module 205 is configured to obtain a data consistency verification result according to the data content consistency verification result.

[0061] In the embodiment of the present application, the data consistency verification device runs the above data consistency verification method. The data consistency verification device divides the verification period according to the number of sampling time periods to obtain a sampling time period, and executes the core query script in the sampling time period according to the number of times of execution of the core script. According to the hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and the data volume consistency verification is performed to obtain the data volume consistency verification result. If the data volume consistency verification result is consistent in data volume, the data content consistency verification is performed on the data of the source database and the target database queried by the core query script to obtain the data content consistency verification result. In this way, the specific number of time points in each time period is selected for sampling verification by hierarchical sampling, thereby reducing the database pressure, discovering data problems in time through periodic verification, improving the effectiveness of verification, and reducing the database pressure.

[0062] Please refer to Figure 3 In the second embodiment of the data consistency verification device in the embodiment of the present application, the data consistency verification device comprises:

[0063] The acquisition module 201 is configured to acquire preset data consistency verification parameters and a core query script. The data consistency verification parameters comprise the number of times of execution of the core script, the verification period and the number of sampling time periods. The core query script is used to query the data of the source database and the target database.

[0064] The query module 202 is configured to divide the verification period according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the number of times of execution of the core script in the sampling time period.

[0065] The data volume verification module 203 is configured to sample the data of the source database and the target database queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and perform data volume consistency verification to obtain a data volume consistency verification result. The N is a natural number not less than 1.

[0066] The data content verification module 204 is configured to, when the data volume consistency verification result is consistent in data volume, perform data content consistency verification on the data of the source database and the target database queried by the core query script to obtain a data content consistency verification result.

[0067] The verification result generation module 205 is configured to obtain a data consistency verification result according to the data content consistency verification result.

[0068] In the embodiment, the data consistency verification device further comprises a first updating module 206. The first updating module 206 is specifically configured to:

[0069] According to the core query script of the core script execution times, the query data amount of data of the source database and the target database is queried, and t sampling time periods with the largest query data amount are screened out from each sampling time period in the verification period, wherein t is a natural number not less than 1;

[0070] According to the hierarchical sampling algorithm, M time points are sampled in each of the t sampling time periods with the largest query data amount, and the data of the source database and the target database queried by the core query script is sampled, wherein M is a natural number not less than N;

[0071] The sampling data of the source database and the target database sampled at the M time points is subjected to data amount consistency verification, and the data amount consistency verification result is subjected to primary updating.

[0072] In the embodiment, the data consistency verification device further comprises a time consumption calculation module 207, which is specifically configured to:

[0073] Obtain the total query time corresponding to the execution of the core query script to query the data of the source database and the target database in each sampling time period;

[0074] According to the total query time corresponding to each sampling time period and the core script execution times, the average time consumption and the proportional time consumption of each sampling time period are calculated.

[0075] In the embodiment, the data consistency verification device further comprises a second updating module 208, which is specifically configured to:

[0076] Screen out t sampling time periods with the highest average time consumption and / or proportional time consumption from each sampling time period of the verification period;

[0077] According to the hierarchical sampling algorithm, W time points are sampled in each of the t sampling time periods with the highest average time consumption and / or proportional time consumption, and the data of the source database and the target database queried by the core query script is sampled, wherein W is a natural number not less than N;

[0078] The sampling data of the source database and the target database sampled at the W time points is subjected to data amount consistency verification, and the data amount consistency verification result is subjected to secondary updating.

[0079] In the embodiment, the data consistency verification parameter further comprises a retry number; the data consistency verification device further comprises a retry module 209, which is specifically configured to:

[0080] If the data volume consistency check result or the data volume consistency check result after one-time update or the data volume consistency check result after two-time update is inconsistent, the data of the source database and the target database is re-queried according to the data consistency check parameter and the core query script after a preset time delay, and data volume consistency check is performed until the data volume consistency check result is consistent or the number of re-queries reaches the retry number.

[0081] In the embodiment, the retry module 209 is specifically further configured to:

[0082] If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry number, a data consistency check result is obtained according to the data volume check result.

[0083] In the embodiment, the data content check module 204 is specifically configured to:

[0084] The data sampling unit 2041 is configured to sample y pieces of data queried by the core query script in each sampling time period from the target database and the source database respectively according to the hierarchical sampling algorithm, wherein y is a natural number not less than 1.

[0085] The content consistency check unit 2042 is configured to perform data content consistency check on the y pieces of sampled data in the target database and the source database, and obtain a data content consistency check result.

[0086] In the embodiment, the content consistency check unit 2042 is specifically configured to:

[0087] The y pieces of sampled data in the target database and the source database are respectively subjected to CRC encoding to obtain corresponding encoding information.

[0088] The data content consistency check is performed on the corresponding encoding information of the source database and the target database to obtain a data content consistency check result.

[0089] In the embodiment, the data consistency check device further includes a third update module 210, and the third update module 210 is specifically configured to:

[0090] The sampled data of the source database and the target database at M time points is subjected to data content consistency check, and the data content consistency check result is subjected to one-time update.

[0091] If the data content consistency check result of the once-updated data content is consistent with the data content of the sampling data of the M time points, the sampling data of the source database and the target database of the W time points are subjected to data content consistency check, and the data content consistency check result is subjected to secondary update.

[0092] In the embodiment, the data consistency check device further comprises a bottom-up compensation module 211, and the retry module 211 is specifically configured to:

[0093] record the data consistency check result in a preset record log, execute a preset automatic execution script, and obtain a corresponding bottom-up compensation script according to the automatic execution script;

[0094] obtain the sampling time period of inconsistent data consistency check according to the bottom-up compensation script, and determine the data segment to be compensated of the target database according to the sampling time period of inconsistent data consistency check;

[0095] obtain the data corresponding to the data segment to be compensated from the source database, and copy the data to the target database.

[0096] The embodiment based on the previous embodiment describes the specific functions of each module and the unit composition of part of the modules. The above modules and units divide the check period according to the number of sampling time periods, obtain the sampling time period, and execute the core query script of the core script execution times in the sampling time period. According to the hierarchical sampling algorithm, the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period, and the data volume consistency check is performed to obtain the data volume consistency check result. If the data volume consistency check result is consistent, the data content consistency check of the data of the source database and the target database queried by the core query script is performed to obtain the data content consistency check result. In this way, the specific number of time points in each time period is selected for sampling verification through hierarchical sampling, thereby reducing the database pressure, discovering data problems in time through periodic check, improving the effectiveness of the check, and reducing the database pressure.

[0097] The above Figure 2 and Figure 3 The data consistency check device in the embodiment of the application is described in detail from the perspective of a modular functional entity, and the data consistency check device in the embodiment of the application is described in detail from the perspective of hardware processing.

[0098] Figure 4Figure 1 is a schematic diagram of an embodiment of a data consistency verification device. The data consistency verification device 400 can include one or more processors (central processing units, CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) storing applications 433 or data 432. The memory 420 and the storage media 430 can be volatile or non-volatile storage. The programs stored in the storage media 430 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the data consistency verification device 400. Further, the processor 410 can be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the data consistency verification device 400 to implement the following steps:

[0099] obtaining preset data consistency verification parameters and a core query script, wherein the data consistency verification parameters include a core script execution number, a verification period, and a number of sampling time periods, and the core query script is used to query data of source and target databases;

[0100] dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the core script execution number in the sampling time period;

[0101] sampling data of the source and target databases queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and performing data volume consistency verification to obtain a data volume consistency verification result, wherein N is a natural number not less than 1;

[0102] if the data volume consistency verification result is data volume consistency, performing data content consistency verification on the data of the source and target databases queried by the core query script to obtain a data content consistency verification result;

[0103] obtaining a data consistency verification result according to the data content consistency verification result.

[0104] Optionally, after the sampling of the data of the source and target databases queried by the core query script at N time points in each sampling time period according to the preset hierarchical sampling algorithm, and the data volume consistency verification to obtain the data volume consistency verification result, the method further includes:

[0105] query data amounts of data of the source database and the target database queried by the core query script according to the core script execution times of each sampling time period, and filter t sampling time periods with the largest query data amounts from each sampling time period in the verification period, wherein t is a natural number not less than 1;

[0106] According to the hierarchical sampling algorithm, M time points are sampled in each of the t sampling time periods with the largest query data amounts to sample data of the source database and the target database queried by the core query script, wherein M is a natural number not less than N;

[0107] The sampled data of the source database and the target database sampled at the M time points are subjected to data amount consistency verification, and the data amount consistency verification result is subjected to primary updating.

[0108] Optionally, if the data amount consistency verification result after the primary updating is that the data amounts of the sampled data sampled at the M time points are consistent, after the data amount consistency verification result is subjected to the primary updating, the method further comprises:

[0109] Obtaining total query time corresponding to querying data of the source database and the target database by the core query script in each sampling time period;

[0110] According to the total query time corresponding to each sampling time period and the core script execution times, average time consumption and proportional time consumption of each sampling time period are calculated.

[0111] Optionally, after the average time consumption and the proportional time consumption of each sampling time period are calculated according to the total query time corresponding to each sampling time period and the core script execution times, the method further comprises:

[0112] Filtering t sampling time periods with the highest average time consumption and / or proportional time consumption from each sampling time period in the verification period;

[0113] According to the hierarchical sampling algorithm, W time points are sampled in each of the t sampling time periods with the highest average time consumption and / or proportional time consumption to sample data of the source database and the target database queried by the core query script, wherein W is a natural number not less than N;

[0114] The sampled data of the source database and the target database sampled at the W time points are subjected to data amount consistency verification, and the data amount consistency verification result is subjected to secondary updating.

[0115] Optionally, the data consistency verification parameter further comprises a retry number;

[0116] After the data volume consistency check according to the preset hierarchical sampling algorithm, the data of the source database and the target database are queried according to the core query script in each sampling time period, and the data volume consistency check is performed to obtain a data volume consistency check result.

[0117] If the data volume consistency check result or the data volume consistency check result after one update or the data volume consistency check result after two updates is inconsistent, after a preset time delay, the data of the source database and the target database are queried according to the data consistency check parameter and the core query script, and the data volume consistency check is performed until the data volume consistency check result is consistent or the number of requerying reaches the retry number.

[0118] Optionally, after the data volume consistency check until the data volume consistency check result is consistent or the number of requerying reaches the retry number, the method further comprises:

[0119] If the data volume consistency check result is inconsistent after the number of requerying reaches the retry number, a data consistency check result is obtained according to the data volume check result.

[0120] Optionally, the data content consistency check according to the data of the source database and the target database queried by the core query script to obtain a data content consistency check result comprises:

[0121] According to the hierarchical sampling algorithm, y pieces of data queried by the core query script in each sampling time period are sampled from the target database and the source database, respectively, wherein y is a natural number not less than 1.

[0122] The y pieces of data sampled from the target database and the source database are subjected to data content consistency check to obtain a data content consistency check result.

[0123] Optionally, the data content consistency check on the y pieces of data sampled from the target database and the source database to obtain a data content consistency check result comprises:

[0124] The y pieces of data sampled from the target database and the source database are subjected to CRC encoding to obtain corresponding encoding information.

[0125] The corresponding encoding information of the source database and the target database is subjected to data content consistency check to obtain a data content consistency check result.

[0126] Optionally, if the data content consistency check result is consistent encoding information, after the y pieces of data sampled from the target database and the source database are subjected to data content consistency check to obtain a data content consistency check result, the method further comprises:

[0127] The sampled data of the source database and the target database at the M time points are subjected to data content consistency check, and the data content consistency check result is subjected to one-time update;

[0128] If the one-time updated data content consistency check result is consistent data content of the sampled data at the M time points, the sampled data of the source database and the target database at the W time points are subjected to data content consistency check, and the data content consistency check result is subjected to two-time update.

[0129] Optionally, if the data consistency check result is inconsistent, after the data consistency check result is obtained according to the data content consistency check result, the method further comprises:

[0130] The data consistency check result is recorded in a preset record log, and a preset automatic execution script is executed, and a corresponding bottom-up compensation script is obtained according to the automatic execution script;

[0131] The sampling time period of inconsistent data consistency check is obtained according to the bottom-up compensation script, and the data segment to be compensated of the target database is determined according to the sampling time period of inconsistent data consistency check;

[0132] The data corresponding to the data segment to be compensated is obtained from the source database and is copied to the target database.

[0133] The data consistency check device 400 can further include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that, Figure 4 The data consistency check device structure shown does not constitute a limitation on the data consistency check device provided by the present application, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0134] The application further provides a computer readable storage medium, which can be a nonvolatile computer readable storage medium or a volatile computer readable storage medium, and the computer readable storage medium stores instructions, and the instructions enable a computer to perform the following steps when the instructions are executed on the computer:

[0135] obtaining preset data consistency verification parameters and a core query script, wherein the data consistency verification parameters include a core script execution frequency, a verification period and a number of sampling time periods, and the core query script is used to query data of source databases and target databases;

[0136] dividing the verification period according to the number of sampling time periods to obtain sampling time periods, and querying the core query script according to the core script execution frequency in the sampling time periods;

[0137] sampling data of the source databases and the target databases queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and performing data volume consistency verification to obtain a data volume consistency verification result, wherein N is a natural number greater than or equal to 1;

[0138] if the data volume consistency verification result is data volume consistency, performing data content consistency verification on the data of the source databases and the target databases queried by the core query script to obtain a data content consistency verification result;

[0139] obtaining a data consistency verification result according to the data content consistency verification result.

[0140] Optionally, after the step of sampling data of the source databases and the target databases queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and performing data volume consistency verification to obtain a data volume consistency verification result, the method further comprises:

[0141] selecting t sampling time periods with the largest query data volume from the sampling time periods in the verification period according to the query data volume of the data of the source databases and the target databases queried by the core query script with the core script execution frequency in each sampling time period, wherein t is a natural number greater than or equal to 1;

[0142] sampling data of the source databases and the target databases queried by the core query script at M time points in each of the t sampling time periods with the largest query data volume according to the hierarchical sampling algorithm, wherein M is a natural number greater than or equal to N;

[0143] The sampled data of the source database and the target database at the M time points is subjected to data volume consistency verification, and the data volume consistency verification result is subjected to primary updating.

[0144] Optionally, if the data volume consistency verification result after the primary updating is that the data volume of the sampled data at the M time points is consistent, after the data volume consistency verification result is subjected to the primary updating, the method further comprises:

[0145] The total query time of the data of the source database and the target database queried by the core query script in each sampling time period is obtained.

[0146] According to the total query time and the core script execution times of each sampling time period, the average time consumption and the proportional time consumption of each sampling time period are calculated.

[0147] Optionally, after the average time consumption and the proportional time consumption of each sampling time period are calculated according to the total query time and the core script execution times of each sampling time period, the method further comprises:

[0148] From the sampling time periods of the verification period, t sampling time periods with the highest average time consumption and / or proportional time consumption are selected.

[0149] According to the hierarchical sampling algorithm, W time points of the data of the source database and the target database queried by the core query script are sampled in each of the t sampling time periods with the highest average time consumption and / or proportional time consumption, wherein W is a natural number not less than N.

[0150] The sampled data of the source database and the target database at the W time points is subjected to data volume consistency verification, and the data volume consistency verification result is subjected to secondary updating.

[0151] Optionally, the data consistency verification parameter further comprises a retry number.

[0152] After the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period according to the preset hierarchical sampling algorithm, and subjected to data volume consistency verification to obtain a data volume consistency verification result, the method further comprises:

[0153] If the data volume consistency verification result or the data volume consistency verification result after the primary updating or the data volume consistency verification result after the secondary updating is inconsistent, after a preset time delay, the data of the source database and the target database is queried again according to the data consistency verification parameter and the core query script, and subjected to data volume consistency verification until the data volume consistency verification result is consistent or the number of times of re-querying reaches the retry number.

[0154] Optionally, after the data volume consistency check is performed until the data volume consistency check result is data volume consistency or the number of re-querying reaches the retry number, the method further comprises:

[0155] If the data volume consistency check result is inconsistency after the number of re-querying reaches the retry number, a data consistency check result is obtained according to the data volume check result.

[0156] Optionally, the data content consistency check on the data of the source database and the target database according to the core query script to obtain a data content consistency check result comprises:

[0157] According to the hierarchical sampling algorithm, y pieces of data queried by the core query script in each sampling time period are sampled from the target database and the source database respectively, wherein y is a natural number not less than 1.

[0158] The y pieces of sampled data in the target database and the source database are subjected to data content consistency check to obtain a data content consistency check result.

[0159] Optionally, the data content consistency check on the y pieces of sampled data in the target database and the source database to obtain a data content consistency check result comprises:

[0160] The y pieces of sampled data in the target database and the source database are subjected to CRC encoding respectively to obtain corresponding encoding information.

[0161] The corresponding encoding information of the source database and the target database is subjected to data content consistency check to obtain a data content consistency check result.

[0162] Optionally, if the data content consistency check result is consistent encoding information, after the data content consistency check on the y pieces of sampled data in the target database and the source database to obtain a data content consistency check result, the method further comprises:

[0163] The sampled data of the source database and the target database at M time points is subjected to data content consistency check, and the data content consistency check result is updated once.

[0164] If the data content consistency check result after the first update is data content consistency of the sampled data at M time points, the sampled data of the source database and the target database at W time points is subjected to data content consistency check, and the data content consistency check result is updated twice.

[0165] Optionally, if the data consistency check result is inconsistent, after obtaining the data consistency check result according to the data content consistency check result, the method further comprises:

[0166] recording the data consistency check result in a preset record log, and executing a preset automatic execution script to obtain a corresponding bottom-up compensation script according to the automatic execution script;

[0167] obtaining a sampling time period of the data consistency check inconsistency according to the bottom-up compensation script, and determining the data segment to be compensated of the target database according to the sampling time period of the data consistency check inconsistency;

[0168] obtaining the data corresponding to the data segment to be compensated from the source database, and copying the data to the target database.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system or device, unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0170] The integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0171] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data consistency checking, the method comprising: The data consistency checking method comprises: obtaining a preset data consistency checking parameter and a core query script, wherein the data consistency checking parameter comprises a core script execution number, a checking period and a sampling time period number, and the core query script is used for querying data of a source database and a target database; dividing the checking period according to the sampling time period number to obtain a sampling time period, and querying the core query script in the sampling time period according to the core script execution number; sampling data of the source database and the target database queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and performing data volume consistency checking to obtain a data volume consistency checking result, wherein N is a natural number not less than 1; if the data volume consistency checking result is data volume consistency, selecting y pieces of data from the data of the source database and the target database queried by the core query script to perform CRC encoding, and performing data content consistency checking based on the encoded information to obtain a data content consistency checking result; obtaining a data consistency checking result according to the data content consistency checking result.

2. The data consistency check method of claim 1, wherein, After the step of sampling data of the source database and the target database queried by the core query script at N time points in each sampling time period according to a preset hierarchical sampling algorithm, and performing data volume consistency checking to obtain a data volume consistency checking result, the method further comprises: selecting t sampling time periods with the largest query data volume from the sampling time periods in the checking period according to the query data volume of the data of the source database and the target database queried by the core query script with the core script execution number in each sampling time period, wherein t is a natural number not less than 1; sampling data of the source database and the target database queried by the core query script at M time points in each of the t sampling time periods with the largest query data volume according to the hierarchical sampling algorithm, wherein M is a natural number not less than N; performing data volume consistency checking on the sampling data of the source database and the target database at the M time points, and updating the data volume consistency checking result once.

3. The data consistency check method of claim 2, wherein, If the data volume consistency checking result after the updating once is data volume consistency of the sampling data at the M time points, the method further comprises, after the step of updating the data volume consistency checking result once: obtaining a total query time corresponding to the data of the source database and the target database queried by the core query script in each sampling time period; calculating average time consumption and proportional time consumption of each sampling time period according to the total query time corresponding to each sampling time period and the core script execution number.

4. The data consistency check method of claim 3, wherein, After the step of calculating average time consumption and proportional time consumption of each sampling time period according to the total query time corresponding to each sampling time period and the core script execution number, the method further comprises: selecting t sampling time periods with the highest average time consumption and / or proportional time consumption from the sampling time periods in the checking period; According to the hierarchical sampling algorithm, W time points in each of t sampling time periods with the highest average time consumption and / or proportional time consumption are used to sample data of the source database and the target database queried by the core query script, wherein W is a natural number not less than N; The sampled data of the source database and the target database at the W time points is subjected to data volume consistency verification, and the data volume consistency verification result is subjected to secondary updating.

5. The data consistency checking method of claim 4, wherein, The data consistency verification parameter further includes a retry number; After the data of the source database and the target database queried by the core query script is sampled at N time points in each sampling time period according to the preset hierarchical sampling algorithm, data volume consistency verification is performed, and a data volume consistency verification result is obtained, the method further includes: If the data volume consistency verification result, the data volume consistency verification result after primary updating or the data volume consistency verification result after secondary updating is inconsistent data volume, after a preset time delay, data of the source database and the target database is re-queried according to the data consistency verification parameter and the core query script, and data volume consistency verification is performed, until the data volume consistency verification result is consistent data volume or the re-querying number reaches the retry number.

6. The data consistency checking method of claim 5, wherein, After the data volume consistency verification is performed until the data volume consistency verification result is consistent data volume or the re-querying number reaches the retry number, the method further includes: If the data volume consistency verification result is inconsistent after the re-querying number reaches the retry number, a data consistency verification result is obtained according to the data volume verification result.

7. The data consistency checking method of claim 3, wherein, The method of selecting y pieces of data from the data of the source database and the target database queried by the core query script to perform CRC encoding, performing data content consistency verification based on the encoded information obtained by encoding, and obtaining a data content consistency verification result includes: According to the hierarchical sampling algorithm, y pieces of data queried by the core query script in each sampling time period are sampled from the target database and the source database respectively, wherein y is a natural number not less than 1; Data content consistency verification is performed on the y pieces of data sampled from the target database and the source database, and a data content consistency verification result is obtained.

8. The data consistency checking method of claim 7, wherein, The method of performing data content consistency verification on the y pieces of data sampled from the target database and the source database, and obtaining a data content consistency verification result includes: The y pieces of data sampled from the target database and the source database are respectively subjected to CRC encoding, and corresponding encoding information is obtained; The corresponding encoding information of the source database and the target database is subjected to data content consistency verification, and a data content consistency verification result is obtained.

9. The data consistency checking method of claim 8, wherein, If the data content consistency verification result is consistent encoding information, after the data content consistency verification is performed on the y pieces of data sampled from the target database and the source database, and the data content consistency verification result is obtained, the method further includes: The sampled data of the source database and the target database at M time points is subjected to data content consistency checking, and the data content consistency checking result is subjected to one-time updating; If the data content consistency checking result after one-time updating is that the data contents of the sampled data at M time points are consistent, the sampled data of the source database and the target database at W time points is subjected to data content consistency checking, and the data content consistency checking result is subjected to two-time updating.

10. The data consistency checking method of claim 1, wherein, If the data consistency checking result is inconsistent, after the data consistency checking result is obtained according to the data content consistency checking result, the method further comprises: The data consistency checking result is recorded in a preset record log, and a preset automatic execution script is executed, and a corresponding bottom-up compensation script is obtained according to the automatic execution script; A sampling time period in which the data consistency checking is inconsistent is obtained according to the bottom-up compensation script, and a data segment to be compensated of the target database is determined according to the sampling time period in which the data consistency checking is inconsistent; The data corresponding to the data segment to be compensated is obtained from the source database and is copied to the target database.

11. A data consistency check apparatus, characterized by comprising: The data consistency checking device comprises: An obtaining module is configured to obtain preset data consistency checking parameters and a core query script, wherein the data consistency checking parameters comprise a core script execution number, a checking period and a number of sampling time periods, and the core query script is used to query data of a source database and a target database; A querying module is configured to divide the checking period according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the core script execution number in the sampling time period; A data volume checking module is configured to take N time points to sample data of the source database and the target database queried by the core query script in each sampling time period according to a preset hierarchical sampling algorithm, and perform data volume consistency checking to obtain a data volume consistency checking result, wherein N is a natural number not less than 1; A data content checking module is configured to select y pieces of data in the data of the source database and the target database queried by the core query script to perform CRC encoding when the data volume consistency checking result is data volume consistent, and perform data content consistency checking based on the encoding information obtained by encoding to obtain a data content consistency checking result; A checking result generating module is configured to obtain a data consistency checking result according to the data content consistency checking result.

12. A data coherency checking device, characterized by The data consistency checking device comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory to enable the data consistency checking device to perform the steps of the data consistency checking method in any one of claims 1-10.

13. A computer-readable storage medium having stored thereon instructions, the instructions comprising, The instructions are executed by the processor to implement the steps of the data consistency checking method in any one of claims 1-10.

Images