Data verification method and device, server and computer readable storage medium

Abstract

This invention provides a data verification method, apparatus, server, and computer-readable storage medium, comprising: acquiring pre-configured source database information and destination database information; wherein the source database is used to store source data, the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data; based on the source database information, the destination database information, and the database logs of the source database, retrieving source sub-data to be verified from the source database and destination sub-data to be verified from the destination database; and performing data verification processing on the source sub-data and destination sub-data using at least one pre-configured target verification thread until the data verification processing is successful and a data verification result is obtained. This invention can significantly enrich the application scenarios of data verification methods, can perform data verification without loss, can effectively reduce the amount of tools required for data verification, and can significantly improve the efficiency of data verification.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data verification method, apparatus, server, and computer-readable storage medium. Background Technology

[0002] MySQL is widely used in various industries such as internet, e-commerce, gaming, and finance. As the concurrency and capacity of single instances increase, database splitting or sharding becomes necessary. Currently, while numerous data migration tools exist, data comparison and verification tools are relatively scarce. Existing data comparison and verification tools have strict verification conditions, requiring the data to have a master-slave relationship, which limits their application scenarios. Furthermore, if data comparison and verification fails, manual re-verification is required, increasing workload and reducing efficiency. Additionally, existing tools perform S-lock operations on business data tables during verification, which can be destructive. Therefore, existing data comparison and verification tools need optimization. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a data verification method, apparatus, server and computer-readable storage medium, which can significantly enrich the application scenarios of data verification methods, can verify data without loss, can effectively reduce the amount of tools required for data verification and significantly improve data verification efficiency.

[0004] In a first aspect, embodiments of the present invention provide a data verification method, comprising: acquiring pre-configured source database information and destination database information; wherein the source database is used to store source data, the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data; based on the source database information, the destination database information, and the database logs of the source database, retrieving source sub-data to be verified from the source database and retrieving destination sub-data to be verified from the destination database; and performing data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing successfully obtains a data verification result.

[0005] Furthermore, the steps of retrieving source sub-data to be verified from the source database and destination sub-data to be verified from the destination database based on the source database information, the destination database information, and the database logs of the source database include: determining the data verification range of the current verification batch based on the database logs of the source database; determining the source sub-data to be verified from the source data and the destination sub-data to be verified from the destination data based on the data verification range of the current verification batch; and retrieving the source sub-data from the source database and the destination sub-data from the destination database based on the source database information.

[0006] Furthermore, the step of determining the data verification range of the current verification batch based on the database log of the source database includes: determining the maximum primary key corresponding to the current log position in the database log of the source database; wherein the current log position is determined based on the data verification range of the previous verification batch; and determining the data verification range of the current verification batch based on the preset number of comparison rows and the maximum primary key.

[0007] Furthermore, the step of using at least one pre-configured target verification thread to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing successfully obtains a data verification result includes: when the data delay between the source database and the destination database is less than a preset delay threshold, using at least one pre-configured target verification thread to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing successfully obtains a data verification result.

[0008] Furthermore, the step of performing data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing successfully obtains a data verification result further includes: establishing at least one candidate verification thread according to a pre-configured number of threads; determining a target verification thread from the candidate verification threads and performing data verification processing on the source sub-data and the destination sub-data using the target verification thread; if the data verification processing fails, determining a next target verification thread from the candidate verification threads and performing data verification processing on the source sub-data and the destination sub-data using the next target verification thread until the data verification processing successfully obtains a data verification result; wherein, the data verification result is used to characterize the consistency of full data and / or incremental data.

[0009] Furthermore, the method also includes: if the data verification process fails, sending the database log to a pre-configured fake slave database to parse the database log and obtain data change information through the fake slave database; and displaying the data change information through a preset display page.

[0010] Furthermore, the method further includes: during the execution of the data verification process, if a control instruction for the data verification process is received, adjusting the working state of the target verification thread according to the control instruction; wherein the control instruction includes a pause instruction and a start instruction, the pause instruction being used to instruct the target verification thread to stop executing the data verification process, and the start instruction being used to instruct the target verification thread to continue executing the data verification process.

[0011] Secondly, embodiments of the present invention also provide a data verification device, comprising: an information acquisition module, configured to acquire pre-configured source database information and destination database information; wherein the source database is used to store source data, the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data; a data retrieval module, configured to retrieve source sub-data to be verified from the source database and destination sub-data to be verified from the destination database based on the source database information, the destination database information, and the database logs of the source database; and a data verification module, configured to perform data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing is successful and a data verification result is obtained.

[0012] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method described in any of the first aspects.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] This invention provides a data verification method, apparatus, server, and computer-readable storage medium. First, pre-configured source database information and destination database information are obtained. The source database stores source data, and the destination database stores destination data obtained by performing database sharding and table partitioning operations on the source data. Then, based on the source database information, destination database information, and the source database's database logs, source sub-data to be verified is retrieved from the source database, and destination sub-data to be verified is retrieved from the destination database. Finally, at least one pre-configured target verification thread performs data verification processing on the source sub-data and destination sub-data until the data verification processing is successful and a data verification result is obtained. The above method can retrieve source and destination sub-data based on pre-configured source database information, target database information, and database logs, and perform data verification accordingly. Compared to existing technologies that require restrictions on master-slave relationships between databases, this invention can more flexibly perform data verification on source and destination databases that do not have a master-slave relationship, thus significantly enriching the application scenarios of data verification methods. Moreover, this invention does not require adding an "S" operation to the source and destination sub-data before using the target verification thread to perform data verification processing, thus enabling lossless data verification processing of the source and destination sub-data. In addition, this invention uses at least one target verification thread to perform data processing operations until the data verification is successful. Compared to existing technologies that require manual secondary comparison and verification, this invention can effectively reduce the amount of tools required for data verification and significantly improve data verification efficiency.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a data verification method provided in an embodiment of the present invention;

[0019] Figure 2 A flowchart illustrating another data verification method provided in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a data verification device provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Currently, the main data comparison and verification tool available is pt-chunksum, provided by MySQL. However, pt-chunksum requires the data to have a master-slave relationship, and the data comparison and verification process adds an S lock to the business data. Therefore, pt-chunksum has the following drawbacks:

[0024] (1) The pt-chunksum tool must satisfy the requirement that the data before and after comparison and verification have a master-slave relationship and the same account and password. However, in actual applications, after the database is split, the source database is MySQL, but the target database is N split databases after the middleware is decoupled, which makes it impossible to meet application scenarios such as database splitting and table migration.

[0025] (2) The pt-chunksum tool adds an S lock during the data table comparison and verification process, which is a detrimental operation to business operations.

[0026] (3) After the first data comparison and verification fails, the pt-chunksum tool can only perform a second comparison and verification manually. In scenarios with a large amount of data, this will result in a heavy workload for the second comparison and verification, and the manual second comparison and verification is inefficient.

[0027] Based on this, the present invention provides a data verification method, apparatus, server, and computer-readable storage medium, which can significantly enrich the application scenarios of data verification methods, verify data without loss, effectively reduce the amount of tools required for data verification, and significantly improve data verification efficiency.

[0028] To facilitate understanding of this embodiment, a data verification method disclosed in this invention will first be described in detail, see [link to relevant documentation]. Figure 1 The diagram shows a data verification method, which mainly includes the following steps S102 to S106:

[0029] Step S102: Obtain pre-configured source database information and destination database information. The source database (hereinafter referred to as the source database) stores source data, and the destination database (hereinafter referred to as the destination database) stores destination data. The destination data is obtained by performing database sharding and table partitioning operations on the source data. The source database information includes one or more of the following: source database name, source table name, source database address, source account, and source password. The destination database information includes one or more of the following: destination database name, destination table name, destination database address, destination account, and destination password.

[0030] In one implementation, a data table stored in the source database can be split into n sub-databases and m sub-tables using a database sharding tool. This operation is called database sharding. The sub-databases are the target databases, and the sub-tables are the target data. Data synchronization is then performed using data migration tools, such as MySQL dump and binlog parsing. Furthermore, this embodiment of the invention also provides an information configuration function. Specifically, it can respond to information configuration operations for the source and target databases to obtain source database information corresponding to the source database and target database information corresponding to the target database.

[0031] Step S104: Based on the source database information, destination database information, and the source database's database log, retrieve the source sub-data to be verified from the source database and the destination sub-data to be verified from the destination database. The database log, also known as the binlog, is a binary log that records all database table structure changes (e.g., CREATE, ALTERTABLE, etc.) and table data modifications (INSERT, UPDATE, DELETE, etc.). In one implementation, the data verification range can be determined based on the database log, thereby determining the source and destination sub-data based on this range, and then retrieving the source and destination sub-data based on the source and destination database information respectively.

[0032] Step S106: At least one pre-configured target verification thread performs data verification processing on the source sub-data and the destination sub-data until the data verification processing is successful and a data verification result is obtained. The data verification result is used to characterize the consistency of the full data and / or the consistency of the incremental data. Full data consistency refers to the consistency based on data at a specific point, while incremental data consistency refers to the consistency compared with database data. In one implementation, a target verification thread can be determined from the candidate verification threads. If the target verification thread successfully performs the data verification processing, a data verification result is obtained; if the target verification thread fails to perform the data verification processing, a next target verification thread is determined from the candidate verification threads, and the aforementioned operation is repeated until the data verification processing is successful and a data verification result is obtained.

[0033] The data verification method provided in this invention can retrieve source and destination sub-data based on pre-configured source database information, target database information, and database logs, and then perform data verification on this basis. Compared with the prior art, which requires restrictions on the master-slave relationship between databases, this invention can more flexibly perform data verification on source and destination databases that do not have a master-slave relationship, thus significantly enriching the application scenarios of the data verification method. Moreover, this invention does not require adding an 'S' operation to the source and destination sub-data before using the target verification thread to perform data verification processing, thus enabling lossless data verification processing of the source and destination sub-data. In addition, this invention uses at least one target verification thread to perform data processing operations until the data verification is successful. Compared with the prior art, which requires manual secondary comparison and verification, this invention can effectively reduce the amount of tools required for data verification and significantly improve data verification efficiency.

[0034] In one embodiment, the data verification method provided by this invention is applied to a server. An information configuration interface is provided on the front end (such as a personal computer, smartphone, or tablet computer). Users can configure relevant information about the source and destination databases on this interface, obtaining source database information such as source database name, source table name, source database address, source account, and source password, as well as destination database information such as destination database name, destination table name, destination database address, destination account, and destination password. In another embodiment, the front end also supports configuring a master-slave relationship between the source and destination databases. It should be noted that even if the source and destination databases are not in a master-slave relationship, the data verification method provided by this invention can still compare and verify the data stored in both, thus significantly enriching the application scenarios of the data verification method. In another embodiment, the front end also supports configuring verification parameters such as the number of concurrent threads and the number of rows compared per thread. In yet another embodiment, the front end also provides a pause / start control. In response to a trigger operation on the pause / start control, the server receives the corresponding control command and controls the working state of the target verification thread according to the control command. This invention supports configuring database information, master-slave relationships, verification parameters, pause / start, etc., making data verification processing more flexible.

[0035] To facilitate understanding of the foregoing embodiments, this invention provides a specific application example of a data verification method, see below. Figure 2 The flowchart of another data verification method shown here mainly includes the following steps S202 to S214:

[0036] Step S202: Obtain source database information and destination database information.

[0037] Step S204: Determine the data verification range of the current verification batch based on the database logs of the source database. To facilitate understanding of step S204, this embodiment of the invention provides an implementation method for determining the data verification range of the current verification batch based on the database logs of the source database, as shown in steps 1 to 2 below:

[0038] Step 1: Determine the maximum primary key corresponding to the current log position in the source database's database log. The current log position (also known as the binlog position) is determined based on the data verification range of the previous verification batch. In practical applications, considering the large amount of source data stored in the source database, this embodiment of the invention proposes multi-batch data verification. In one implementation, when determining the data verification range of the previous verification batch, the maximum primary key of the previous verification batch can be obtained. This maximum primary key can be used in the current verification batch; this maximum primary key is also the maximum primary key corresponding to the current log position.

[0039] In one implementation, the current log position can be obtained by executing the `show master status` command in the current verification batch. By obtaining the current log position, the accuracy of all data before the current log position and the accuracy of incremental data can be ensured. At the same time, the binlog after the current log position can be verified by rolling the current log position.

[0040] Step 2: Determine the data verification range of the current verification batch based on the preset number of comparison rows and the maximum primary key. The preset number of comparison rows can be configured through the front end. In one implementation, the data verification range of the current verification batch can be determined using the primary key / unique key and the preset number of comparison rows. For example, the `limit` directive can be used to determine the data verification range, such as: `primary key >= minimum primary key limit preset number of comparison rows`.

[0041] Step S206: Based on the data verification range of the current verification batch, determine the source sub-data to be verified from the source data and the destination sub-data to be verified from the destination data.

[0042] In one implementation, the data verification range is equivalent to defining the start and end points in the source or destination data. Data in the source data that matches the data verification range can be directly used as the source sub-data to be verified. Similarly, data in the destination data that matches the data verification range can be directly used as the destination sub-data to be verified.

[0043] Step S208: Retrieve source sub-data from the source database and retrieve destination sub-data from the destination database based on the source database information.

[0044] In one implementation, the source database can be logged into using the source account and source password, and source sub-data can be retrieved from the source database address while logged in; similarly, the target database can be logged into using the target account and target password, and target sub-data can be retrieved from the target database address while logged in.

[0045] In one implementation, considering that excessive master-slave latency between the source and destination databases can lead to significant data inconsistencies, and that lower latency significantly reduces the amount of data requiring secondary verification, the data latency between the source and destination databases can be pre-judged before data verification processing. If the latency is less than a preset latency threshold, at least one pre-configured target verification thread performs data verification processing on the source and destination sub-data until the verification is successful and a data verification result is obtained. This embodiment also provides a specific implementation method for data verification processing, as shown in steps S210 to S214 below.

[0046] Step S210: Create at least one candidate verification thread based on the pre-configured number of threads. The number of threads can be configured via the front end, and the candidate verification thread can be referred to as a candidate job thread.

[0047] In one implementation, the front end can provide a thread configuration control, which can be an input box control or a list selection control. In response to an operation on the thread configuration control, the front end can determine the number of threads configured by the user and create a corresponding number of candidate verification threads based on the number of threads.

[0048] Step S212: Determine the target verification thread from the candidate verification threads, and use the target verification thread to perform data verification processing on the source sub-data and the destination sub-data.

[0049] In one implementation, a verification thread can be randomly selected from the candidate verification threads as the target verification thread, and the target verification thread can be used to perform data verification processing.

[0050] Step S214: If the data verification process fails, determine the next target verification thread from the candidate verification threads, and use the next target verification thread to perform data verification processing on the source sub-data and the destination sub-data until the data verification process is successful and a data verification result is obtained. The data verification result is used to characterize the consistency of the full data and / or the consistency of the incremental data. For example, assuming the candidate verification threads include job1, job2, and job3, job1 is randomly selected as the target verification thread. When job1 fails to perform data verification processing, job3 is randomly selected as the target verification thread, so that job2 performs a second / Nth data verification process until the data verification process is successful.

[0051] In another implementation, if data verification fails, the database logs are sent to a pre-configured fake slave database. The fake slave database parses the database logs to obtain data change information, which is then displayed on a pre-defined page. In one implementation, the fake slave database can be something like Canal. The fake slave database disguises the target database using Canal to obtain database logs from the source database. Canal parses the database logs to ensure that data verification failures occur in records modified in the database logs, further ensuring data integrity. It also performs secondary / Nth data verification processes on the changed data to guarantee successful data verification.

[0052] Optionally, embodiments of the present invention also provide visualization display functions, such as displaying binlog sites, displaying or querying detailed data on binlog data changes, etc.

[0053] In another implementation, during the data verification process, if a control command for data verification is received, the working state of the target verification thread is adjusted according to the control command. The control command includes a pause command and a start command. The pause command instructs the target verification thread to stop executing data verification processing, and the start command instructs the target verification thread to continue executing data verification processing. The working state is thus either paused or resumed data verification processing. Furthermore, the interface may also support a pause / start control, responding to user actions (such as clicks, swipes, gestures, etc.) to generate corresponding control commands, thereby controlling whether the target verification thread continues executing data verification processing based on these commands.

[0054] In summary, the data verification method provided in this invention proposes lossless full and incremental data verification before and after MySQL database sharding and table migration, which can significantly improve the accuracy and security of data before and after migration. Furthermore, the data verification method provided in this invention is perfectly compatible with various business migration scenarios. Compared with the existing pt-chunksum tool, this invention has the advantage of more flexibility in covering more application scenarios, meeting the ever-changing needs of business, and better supporting business operations. In addition, the data verification method provided in this invention does not add locks during the data verification process, thus enabling lossless data verification processing, ensuring no impact on business operations, better meeting business needs, and greatly reducing the impact of data verification processing on business operations.

[0055] Regarding the data verification method provided in the foregoing embodiments, this invention provides a data verification device, see [link to relevant documentation]. Figure 3 The diagram shows the structure of a data verification device, which mainly includes the following parts:

[0056] The information acquisition module 302 is used to acquire pre-configured source database information and destination database information; wherein, the source database is used to store source data, the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data;

[0057] The data retrieval module 304 is used to retrieve source sub-data to be verified from the source database and destination sub-data to be verified from the destination database, based on source database information, destination database information and database logs of the source database.

[0058] The data verification module 306 is used to perform data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing is successful and the data verification result is obtained.

[0059] The data verification device provided in this invention can retrieve source and destination sub-data based on pre-configured source database information, target database information, and database logs, and perform data verification accordingly. Compared to existing technologies that require restrictions on the master-slave relationship between databases, this invention can more flexibly perform data verification on source and destination databases that do not have a master-slave relationship, thus significantly enriching the application scenarios of the data verification method. Moreover, this invention does not require adding an "S" operation to the source and destination sub-data before using the target verification thread to perform data verification processing, thus enabling lossless data verification processing of the source and destination sub-data. In addition, this invention uses at least one target verification thread to perform data processing operations until the data verification is successful. Compared to existing technologies that require manual secondary comparison and verification, this invention can effectively reduce the amount of tools required for data verification and significantly improve data verification efficiency.

[0060] In one implementation, the data retrieval module 304 is further configured to: determine the data verification range of the current verification batch based on the database logs of the source database; determine the source sub-data to be verified from the source data and the destination sub-data to be verified from the destination data based on the data verification range of the current verification batch; and retrieve the source sub-data from the source database and the destination sub-data from the destination database based on the source database information.

[0061] In one implementation, the data retrieval module 304 is further configured to: determine the maximum primary key corresponding to the current log position in the database log of the source database; wherein the current log position is determined based on the data verification range of the previous verification batch; and determine the data verification range of the current verification batch based on the preset number of comparison rows and the maximum primary key.

[0062] In one embodiment, the data verification module 306 is further configured to: when the data delay between the source database and the destination database is less than a preset delay threshold, use at least one pre-configured target verification thread to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing is successful and the data verification result is obtained.

[0063] In one implementation, the data verification module 306 is further configured to: establish at least one candidate verification thread according to a pre-configured number of threads; determine a target verification thread from the candidate verification threads, and use the target verification thread to perform data verification processing on the source sub-data and the destination sub-data; if the data verification processing fails, determine the next target verification thread from the candidate verification threads, and use the next target verification thread to perform data verification processing on the source sub-data and the destination sub-data, until the data verification processing is successful and a data verification result is obtained; wherein, the data verification result is used to characterize the consistency of full data and / or the consistency of incremental data.

[0064] In one embodiment, the above-mentioned device further includes a log parsing module, used to: if the data verification process fails, send the database log to a pre-configured fake slave database to obtain data change information by parsing the database log through the fake slave database; and display the data change information through a preset display page.

[0065] In one embodiment, the above-mentioned device further includes a thread control module, configured to: during the execution of data verification processing, if a control instruction for data verification processing is received, adjust the working state of the target verification thread according to the control instruction; wherein the control instruction includes a pause instruction and a start instruction, the pause instruction being used to instruct the target verification thread to stop executing data verification processing, and the start instruction being used to instruct the target verification thread to continue executing data verification processing.

[0066] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0067] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0068] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0069] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0070] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0071] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0072] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0073] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0074] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data checking method, characterized by, include: Obtain pre-configured source database information and destination database information; wherein, the source database is used to store source data, the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data; Based on the source database information, the destination database information, and the database logs of the source database, the source sub-data to be verified is retrieved from the source database, and the destination sub-data to be verified is retrieved from the destination database, respectively. At least one pre-configured target verification thread is used to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing is successful and a data verification result is obtained. The step of performing data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing successfully obtains a data verification result further includes: At least one candidate verification thread is created based on the pre-configured number of threads; A target verification thread is determined from the candidate verification threads, and the target verification thread is used to perform data verification processing on the source sub-data and the destination sub-data; If the data verification process fails, a next target verification thread is determined from the candidate verification threads, and the next target verification thread is used to perform data verification processing on the source sub-data and the destination sub-data until the data verification process is successful and a data verification result is obtained; wherein, the data verification result is used to characterize the consistency of full data and / or incremental data.

2. The method of claim 1, wherein, The steps of retrieving source sub-data to be verified from the source database and destination sub-data to be verified from the destination database based on the source database information, the destination database information, and the database logs of the source database include: The data verification range for the current verification batch is determined based on the database logs of the source database. Based on the data verification range of the current verification batch, determine the source sub-data to be verified from the source data, and determine the destination sub-data to be verified from the destination data; The source sub-data is retrieved from the source database based on the source database information, and the destination sub-data is retrieved from the destination database.

3. The method of claim 2, wherein, The step of determining the data verification range of the current verification batch based on the database logs of the source database includes: Determine the maximum primary key corresponding to the current log position in the database log of the source database; wherein the current log position is determined based on the data verification range of the previous verification batch; Based on the preset number of rows to be compared and the maximum primary key, the data verification range of the current verification batch is determined.

4. The method of claim 1, wherein, The step of using at least one pre-configured target verification thread to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing is successful and a data verification result is obtained includes: If the data delay between the source database and the destination database is less than a preset delay threshold, at least one pre-configured target verification thread is used to perform data verification processing on the source sub-data and the destination sub-data until the data verification processing is successful and a data verification result is obtained.

5. The method of claim 1, wherein, The method further includes: If the data verification process fails, the database log is sent to a pre-configured fake slave database so that the fake slave database can parse the database log to obtain data change information. The data change information is displayed on a preset display page.

6. The method of claim 1, wherein, The method further includes: During the execution of the data verification process, if a control instruction for the data verification process is received, the working state of the target verification thread is adjusted according to the control instruction. The control instructions include a pause instruction and a start instruction. The pause instruction is used to instruct the target verification thread to stop executing the data verification process, and the start instruction is used to instruct the target verification thread to continue executing the data verification process.

7. A data verification device, characterized in that, include: The information acquisition module is used to acquire pre-configured source database information and destination database information; wherein, the source database is used to store source data, and the destination database is used to store destination data, and the destination data is obtained by performing database sharding and table partitioning operations on the source data; The data retrieval module is used to retrieve source sub-data to be verified from the source database and target sub-data to be verified from the target database, based on the source database information, the target database information, and the database logs of the source database. The data verification module is used to perform data verification processing on the source sub-data and the destination sub-data using at least one pre-configured target verification thread until the data verification processing is successful and a data verification result is obtained. The data verification module is further configured to: establish at least one candidate verification thread based on a pre-configured number of threads; determine a target verification thread from the candidate verification threads, and use the target verification thread to perform data verification processing on the source sub-data and the destination sub-data; if the data verification processing fails, determine a next target verification thread from the candidate verification threads, and use the next target verification thread to perform data verification processing on the source sub-data and the destination sub-data, until the data verification processing is successful and a data verification result is obtained; wherein the data verification result is used to characterize the consistency of full data and / or incremental data.

8. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images