Data verification method, storage medium and device

Abstract

The present application relates to the field of database, in particular to a data verification method, a storage medium and equipment. The data verification method comprises obtaining a plurality of tables to be verified and parameter information of each table to be verified; processing the plurality of tables to be verified according to the parameter information of each table to be verified, and generating a sequential queue to be verified; and performing a data verification operation on the sequential queue to be verified. The data verification method of the present application sorts the plurality of tables to be verified in advance according to the parameter information of each table to be verified, realizes batch comparison according to the importance of the tables to be verified in the process of data verification, avoids the problem that the verification result of important data is not timely output, and guarantees the timeliness and accuracy of data verification.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a data verification method, storage medium, and device. Background Technology

[0002] Real-time data synchronization using data synchronization software is divided into three stages: The first stage initializes and loads existing data to establish a baseline for data synchronization; the second stage uses this baseline as a benchmark for incremental data synchronization; and the third stage periodically compares and verifies the source and target data to ensure no data is lost during the synchronization process. The second and third stages typically run in parallel for extended periods.

[0003] In the second phase of incremental data synchronization, a common real-time data replication technique involves analyzing database logs to obtain incremental data, thereby achieving real-time data synchronization. This technique parses the online or archived logs of the source database to obtain the data additions, deletions, and changes. These changes are then converted into a specific message format within the synchronization software, unit by transaction, and sent to the target synchronization software via its proprietary transmission protocol. Finally, the target synchronization software reconstructs the obtained transaction logs into SQL statements supported by the target database and executes them on the target database to achieve real-time data synchronization and maintain data consistency between the source and target databases.

[0004] In the third phase of the operation, the verification method typically involves querying data from a table in both the source and target databases and storing it in memory, then performing verification using a specific algorithm in memory. This process is repeated until all tables to be compared have been compared.

[0005] However, while existing data validation methods can functionally meet the requirements for table data validation, the number of tables and records being validated gradually increases with business growth. Furthermore, due to user business needs, some business tables lack primary keys, and the frequent use of these tables can result in some tables having excessively large data volumes (typically reaching hundreds of gigabytes). The existence of these tables without primary keys and with excessively large data volumes significantly increases the time consumed in the data validation process. Users often do not focus on the data consistency status of these tables without primary keys and with large data volumes, leading to delays in obtaining the data discrepancies of the tables they are interested in, thus compromising the timeliness or accuracy of data validation. Summary of the Invention

[0006] One object of the present invention is to provide a data verification method, storage medium and device that can solve any of the above problems.

[0007] A further objective of this invention is to improve the timeliness or accuracy of data verification.

[0008] Another further objective of this invention is to improve the efficiency of data verification.

[0009] Specifically, the present invention provides a data verification method, which includes:

[0010] Obtain multiple tables to be verified and parameter information for each table;

[0011] The multiple tables to be verified are processed based on the parameter information of each table, and a sequential queue of tables to be verified is generated.

[0012] Perform data verification operations on the sequential queue to be verified.

[0013] Optionally, the parameter information of the table to be verified includes the data volume of the table, attribute information indicating whether the table has a primary key, and identifier information indicating whether the table needs to be sharded; and

[0014] The steps for processing multiple tables to be verified based on their parameter information and generating a sequential queue of tables to be verified include:

[0015] Multiple tables to be verified are sorted according to the amount of data and attribute information of each table to be verified in order to generate a priority queue;

[0016] Obtain the checklist that needs to be sharded in the priority queue based on the identification information;

[0017] The table to be verified that needs to be sharded is sharded, and the sharded table to be verified is moved to the tail of the priority queue to generate a sequential queue to be verified.

[0018] Optionally, the step of sorting multiple tables to be verified based on the data volume and attribute information of each table includes:

[0019] Based on the data volume and attribute information of each table to be verified, mark the priority of each table to be verified;

[0020] The multiple tables to be verified are sorted according to their priority.

[0021] Optionally, the step of prioritizing each table to be verified based on its data volume and attribute information includes:

[0022] The data volume of the table to be verified is compared with a preset data volume threshold, and the priority of the table to be verified is adjusted based on the comparison result; and

[0023] Determine whether the table to be verified has a primary key based on the attribute information;

[0024] If not, then lower the priority of the table to be verified.

[0025] Optionally, the steps of sharding the table to be verified that requires sharding and moving the sharded table to the tail of the priority queue to generate a sequential queue to be verified include:

[0026] The table to be verified that needs to be sharded is sharded using a preset sharding algorithm to obtain multiple smaller tables;

[0027] Remove the tables to be verified that need to be sharded from the priority queue;

[0028] Multiple smaller tables are sequentially placed at the tail of a priority queue, and each smaller table is treated as a table to be verified, thus generating a sequential queue to be verified.

[0029] Optionally, the step of partitioning the table to be verified using a preset partitioning algorithm to obtain multiple smaller tables includes:

[0030] Obtain the preset sharding baseline corresponding to the verification table that needs to be sharded;

[0031] Based on the sharding criteria, the table to be verified that needs to be sharded is sharded to obtain multiple smaller tables.

[0032] Optionally, the steps for performing data verification operations on the sequential queue to be verified include:

[0033] Based on the preset number of parallel operations, the same number of tables to be verified are sequentially retrieved from the sequential queue to be verified.

[0034] Parallel comparisons are performed on each group of tables to be verified, with the same number of tables as the parallel number.

[0035] Optionally, after performing data verification operations on the sequential queue to be verified, the data verification method further includes:

[0036] The verification results are stored in the metadata database.

[0037] According to another aspect of the present invention, a machine-readable storage medium is also provided, on which a machine-executable program is stored, wherein the machine-executable program, when executed by a processor, implements any of the above-described data verification methods.

[0038] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor implements any of the above-described data verification methods when executing the machine-executable program.

[0039] The data verification method of this invention obtains multiple tables to be verified and parameter information of each table, processes the multiple tables to be verified according to the parameter information of each table, generates a sequential queue of tables to be verified, and performs data verification operations on the sequential queue of tables to be verified. This achieves the pre-sorting of multiple tables to be verified according to the parameter information of each table, so that during the data verification process, the tables to be verified can be compared in batches according to their importance, ensuring the timeliness and accuracy of data verification, thereby improving the synchronization performance of the data synchronization software.

[0040] The data verification method of this invention uses parameter information for the tables to be verified, including the data volume of the table, attribute information indicating whether the table has a primary key, and identifier information indicating whether the table needs to be sharded. Furthermore, this invention first sorts multiple tables to be verified based on their data volume and attribute information to generate a priority queue. Then, it retrieves the tables to be verified that need sharding from the priority queue based on the identifier information, performs sharding on these tables, and after sharding, performs a second sorting on all tables to be verified, thus generating a sequential queue for verification. This data verification method, through the first sorting, ensures that important data of concern to the user is verified first, and data that is not of concern or importance to the user is verified later, further guaranteeing the timeliness and accuracy of data verification. Furthermore, through sharding, a large table to be verified is divided into multiple smaller tables for parallel comparison, greatly improving the comparison efficiency of a single large table to be verified, thereby further improving the synchronization performance of the data synchronization software.

[0041] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0042] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 This is a schematic flowchart of a data verification method according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram illustrating an application scenario of a data verification method according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating a data verification method according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a machine-readable storage medium according to an embodiment of the present invention; and

[0047] Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0048] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0049] To address the aforementioned technical problems, this invention proposes a method for processing write-ahead logs. Figure 1 This is a schematic flowchart of a data verification method according to an embodiment of the present invention. Figure 2 A schematic diagram illustrating an application scenario of a data verification method according to an embodiment of the present invention. For example... Figure 1 As shown, the data verification method in this embodiment generally includes:

[0050] Step S102: Obtain multiple tables to be verified and parameter information for each table.

[0051] Step S104: Process multiple tables to be verified according to the parameter information of each table to be verified, and generate a sequential queue to be verified.

[0052] Step S106: Perform a data verification operation on the sequential queue to be verified.

[0053] In step S102 above, the multiple tables to be verified can be all tables pre-selected and determined by the user. The parameter information for each table to be verified includes the data volume of the table, attribute information indicating whether the table has a primary key, and identifier information indicating whether the table needs to be sharded. Specifically, the data volume and attribute information of the tables to be verified are parameter information only related to the data of the tables to be verified, and can all be obtained sequentially through JDBC. For example... Figure 2As shown, the database tables to be validated may include tables A, B, C, D, and E. Table A has 1 million rows, table B has 2 million rows, table C has 3 million rows, table D has 20,000 rows, and table E has 10,000 rows. Tables A, D, and E are tables with primary keys, while tables B and C are tables without primary keys. The identification information of the tables to be validated can be obtained by the user's pre-selection of the tables to be sharded and is pre-stored in the database. Furthermore, the data volume, attribute information, and identification information of each table to be validated correspond one-to-one with the tables to be validated.

[0054] In step S104 above, the sequential queue to be verified is a queue composed of multiple tables to be verified arranged in a certain order.

[0055] Step S106 above may include the following steps: according to a preset parallel number, sequentially taking out the same number of tables to be verified from the sequential queue to be verified, and performing parallel comparison on each group of tables to be verified with the same number of parallel numbers. That is, in step S106 above, the verification executor first takes out a specified number of tables to be verified from the sequential queue to be verified, where the specified number is the preset parallel number, and then performs data verification on the specified number of tables to be verified.

[0056] In addition, after step S106 above, the data verification method of this embodiment may further include the following step: storing the verification result in a metadata database. Specifically, after completing the data verification for each table to be verified, the verification result is written to the metadata database to complete the data verification process.

[0057] The data verification method in this embodiment obtains multiple tables to be verified and their parameter information, processes the multiple tables to be verified according to their parameter information, generates a sequential queue of tables to be verified, and performs data verification operations on the sequential queue of tables to be verified. This achieves the pre-sorting of multiple tables to be verified according to their parameter information, so that during the data verification process, the tables to be verified can be compared in batches according to their importance, ensuring the timeliness and accuracy of data verification, thereby improving the synchronization performance of the data synchronization software.

[0058] In some embodiments, step S104 may include the following steps: sorting multiple tables to be verified according to the data volume and attribute information of each table to be verified to generate a priority queue; obtaining the tables to be verified that need to be sharded in the priority queue according to the identification information; performing sharding processing on the tables to be verified that need to be sharded, and moving the sharded tables to be verified to the tail of the priority queue to generate a sequential queue to be verified. That is, in step S104, the data verification method of this embodiment performs two sorting operations. First, the data verification method of this embodiment performs a first sorting of multiple tables to be verified according to the data volume and attribute information of each table to be verified, generating a priority queue. Specifically, as shown... Figure 2 As shown, according to the preset priority algorithm, in the priority queue consisting of tables A, B, C, D, and E, the order of multiple tables to be verified can be from back to front as table C, table B, table A, table D, and table E. Secondly, in this embodiment, after sharding the tables to be verified that require sharding in the priority queue, the data verification method performs a second sorting on all tables to be verified, splitting the tables with larger data volumes into multiple smaller tables and placing them at the end of the queue, thus obtaining the final sequential queue of tables to be verified.

[0059] The data verification method in this embodiment first sorts the data to ensure that important data of concern to the user is verified first, and data that is not of concern or importance to the user is verified later. This further ensures the timeliness and accuracy of data verification. In addition, by using sharding, a large data table to be verified is divided into multiple smaller data tables to be verified and compared in parallel. This greatly improves the comparison efficiency of a single large data table to be verified, thereby further improving the synchronization performance of the data synchronization software.

[0060] In some embodiments, the step of sorting multiple tables to be verified based on the data volume and attribute information of each table to be verified may specifically include the following steps: marking the priority of each table to be verified based on the data volume and attribute information of each table to be verified; and sorting the multiple tables to be verified based on the priority of each table to be verified. That is, the first sorting in step S104 above uses a preset priority algorithm, assigning a priority to each table to be verified based on its data volume and attribute information. In other words, all tables to be verified are marked with a priority attribute. Furthermore, in the step of sorting the multiple tables to be verified based on their priorities, tables with higher priorities are ranked first, and tables with lower priorities are ranked last, thus generating a priority queue.

[0061] The data verification method in this embodiment performs an initial sorting of all tables to be verified based on a preset priority algorithm. This ensures, to a certain extent, that important data that users may care about is placed first, further guaranteeing the timeliness and accuracy of data verification.

[0062] In some embodiments, the step of marking the priority of each table to be verified based on the data volume and attribute information of each table to be verified may include the following steps: comparing the data volume of the table to be verified with a preset data volume threshold, and adjusting the priority of the table to be verified based on the comparison result; and determining whether the table to be verified has a primary key based on the attribute information; if not, reducing the priority of the table to be verified.

[0063] Specifically, the aforementioned preset data volume threshold can be determined based on the average data volume of the tables to be verified or obtained through prior experiments. The step of comparing the data volume of the tables to be verified with the preset data volume threshold and adjusting the priority of the tables to be verified based on the comparison results can be implemented as follows: First, record the priority of all tables to be verified as 0. Then, adjust the priority of each table to be verified based on its data volume. For every additional 1 million rows of data in each table to be verified, record the priority of that table as -1. Using this method, multiple tables to be verified are sorted in ascending order of data volume, thus enabling priority verification of tables with smaller data volumes during batch comparisons.

[0064] Furthermore, when the table to be verified has no primary key, the aforementioned step of reducing the priority of the table to be verified can be specifically executed as follows: multiply the priority of the table to be verified without a primary key by 2. Using the above method, multiple tables to be verified are sorted in the order of those with primary keys first, followed by those without primary keys, thus ensuring that tables with primary keys are verified first during batch comparison.

[0065] like Figure 2 As shown, tables A, B, C, D, and E are sorted for the first time using a preset priority algorithm. First, the priority of tables A, B, C, D, and E is recorded as 0. Next, for every additional 1 million rows in each table to be verified, the priority of that table is decremented by 1, resulting in a priority of -1 for table A, -2 for table B, -3 for table C, and 0 for table D. Then, the priority of the tables without primary keys is multiplied by 2, resulting in a priority of -4 for table B and -6 for table C. This completes the priority calculation for tables A through E. Finally, tables A through E are sorted according to their priority, resulting in tables E, D, A, B, and C arranged from highest to lowest priority.

[0066] In some embodiments, the steps of sharding the table to be verified that needs sharding and moving the sharded table to the tail of the priority queue to generate a sequential queue for verification may include the following steps: sharding the table to be verified that needs sharding using a preset sharding algorithm to obtain multiple smaller tables; removing the table to be verified that needs sharding from the first sequential queue; and sequentially placing the multiple smaller tables into the tail of the first sequential queue, with each smaller table serving as a table to be verified, to generate a sequential queue for verification. That is, the second sorting in step S104 above uses a preset sharding algorithm to shard the table to be verified that the user specifies as to be sharded according to the sharding column.

[0067] The data verification method in this embodiment uses a sharding algorithm to divide a large table into multiple smaller tables for parallel comparison, greatly improving the comparison efficiency of a single large table. In subsequent steps, the multiple smaller tables obtained from the sharding process are sequentially placed at the end of a priority queue, achieving a second sorting of all tables to be verified. This ensures that tables with large amounts of data that users are not concerned about or that are not important are placed at the end, thereby further guaranteeing the timeliness and accuracy of data verification.

[0068] Furthermore, the above-mentioned step of sharding the table to be sharded using a preset sharding algorithm to obtain multiple smaller tables can be specifically executed as follows: obtaining a preset sharding benchmark corresponding to the table to be sharded; and sharding the table to be sharded according to the sharding benchmark to obtain multiple smaller tables. Specifically, the sharding benchmark corresponding to the table to be sharded can be obtained by the user selecting specific columns in the specified table to be sharded and pre-stored in the database.

[0069] In one specific embodiment, the specific method for sharding the table to be verified that needs to be sharded according to the sharding benchmark may include: when the column type of the table to be verified that needs to be sharded is numeric, first sort the values ​​of the column of the table to be verified that needs to be sharded in ascending order, and then perform a modulo operation on the minimum and maximum values, that is, divide the data within the range into a certain number of batches, with each batch containing approximately the same amount of data, so as to complete the sharding process.

[0070] In another specific embodiment, the specific method for sharding the table to be verified that needs to be sharded according to the sharding benchmark may also include: when the column type of the table to be verified that needs to be sharded is time type, first sort the values ​​of the column of the table to be verified that needs to be sharded in ascending order, and then perform a modulo operation with the minimum and maximum values, that is, divide the data within the range into a certain number of batches, with each batch containing approximately the same amount of data, so as to complete the sharding process.

[0071] In another specific embodiment, the specific method for sharding the table to be verified that needs to be sharded according to the sharding benchmark may also include: when the column type is character type, the MD5 value of the character needs to be calculated first, and then its MD5 value is converted to hexadecimal. After being converted to hexadecimal, it is sharded by taking the modulus using a specific method similar to that for number type or time type.

[0072] After dividing the table to be verified into specified shards, each shard is treated as a separate table to be verified and placed at the end of the priority queue. The original tables of the tables to be verified that previously needed to be sharded are removed from the priority queue, thus splitting a large table into multiple smaller tables and completing the second sorting.

[0073] Using the above method, multiple tables to be verified were processed, resulting in the final sequential queue of tables to be verified. This enabled post-verification of tables with large amounts of data and splitting them into multiple smaller tables for parallel comparison, greatly improving the comparison efficiency of a single large table and thus further enhancing the synchronization performance of the data synchronization software.

[0074] Figure 3 This is a flowchart illustrating a data verification method according to an embodiment of the present invention. The following is in conjunction with... Figure 3 The process steps of this embodiment will be described in detail.

[0075] Step S302: Provide external configuration parameters and determine the data verification method in response to the user's configuration.

[0076] Step S304: Determine whether to perform batch comparison operation. If yes, proceed to step S306; otherwise, proceed to step S330.

[0077] Step S306: In response to the user's selection operation, store all the tables to be verified selected by the user and the parameter information of each table to be verified in the database. Specifically, the parameter information of each table to be verified includes the data volume of the table to be verified, attribute information used to identify whether the table to be verified has a primary key, identification information used to identify whether the table to be verified needs to be sharded, and the sharding benchmark corresponding to the table to be verified that needs to be sharded.

[0078] Step S308: Obtain multiple tables to be verified and the amount of data in each table.

[0079] Step S310: Compare the data volume of the table to be verified with a preset data volume threshold, and adjust the priority of the table to be verified based on the comparison result. Specifically, this step can be executed as follows: first, record the priority of all tables to be verified as 0, then adjust the priority of each table to be verified based on the data volume of each table to be verified, and record the priority of each table to be verified -1 for every additional 1 million rows of data.

[0080] Step S312: Obtain the attribute information of each table to be verified.

[0081] Step S314: Determine whether the table to be verified has a primary key based on the attribute information. If it does, proceed to step S316; otherwise, proceed to step S318.

[0082] Step S316: Do not change the priority of the table to be verified, and proceed to step S320.

[0083] Step S318: Reduce the priority of the table to be verified and proceed to step S320.

[0084] Step S320: Sort the multiple tables to be verified according to their priority to generate a priority queue. The specific sorting method is as follows: higher priority tables are listed first, and lower priority tables are listed last.

[0085] Step S322: Obtain the check table that needs to be sharded in the priority queue based on the identification information.

[0086] Step S324: The table to be verified that needs to be sharded is sharded using a preset sharding algorithm to obtain multiple smaller tables. Specifically, this step can be performed as follows: obtaining a preset sharding benchmark corresponding to the table to be verified that needs to be sharded; and sharding the table to be verified that needs to be sharded according to the sharding benchmark to obtain multiple smaller tables.

[0087] Step S326: Remove the tables to be verified that need to be sharded from the priority queue, put multiple small tables into the tail of the priority queue in sequence, and treat each small table as a table to be verified to generate a sequential queue to be verified.

[0088] Step S328: Perform data verification operation on the sequential queue to be verified, and then proceed to step S332. Specifically, this step can be performed as follows: according to the preset parallel number, sequentially retrieve the same number of tables to be verified from the sequential queue to be verified; perform parallel comparison on each group of tables to be verified with the same number of parallel numbers.

[0089] Step S330: Skip the batch comparison operation, sequentially obtain data tables for data verification, and then proceed to step S332. Specifically, the step of sequentially obtaining data tables for data verification can be executed by querying a data table from both the source and target databases into memory, and performing verification in memory using a specific algorithm. This process is repeated until all data tables requiring verification have been compared.

[0090] Step S332: Store the verification result in the metadata database. This concludes the current process.

[0091] Through the above steps, the data verification method of this embodiment can sort multiple tables to be verified in advance according to the parameter information of each table to be verified. Thus, during the data verification process, the tables to be verified can be compared in batches according to their importance, ensuring the timeliness and accuracy of data verification, thereby improving the synchronization performance of the data synchronization software.

[0092] The present invention also provides a machine-readable storage medium and a computer device. Figure 4 This is a schematic diagram of the structure of a machine-readable storage medium 10 according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a computer device 20 according to an embodiment of the present invention.

[0093] The machine-readable storage medium 10 stores a machine-executable program 11 thereon, which, when executed by a processor, implements the processing method of any of the above embodiments.

[0094] The computer device 20 may include a memory 220, a processor 210, and a machine-executable program 11 stored on the memory 220 and running on the processor 210, and the processor 210 implements the processing method of any of the above embodiments when executing the machine-executable program 11.

[0095] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-based system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0096] For the purposes of this embodiment, the machine-readable storage medium 10 can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection (electronic device) having one or more wires, a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable medium 10 can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0097] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0098] Computer device 20 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 20 can be a cloud computing node. Computer device 20 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 20 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0099] Computer device 20 may include a processor 210 adapted to execute stored instructions and a memory 220 that provides temporary storage space for the operation of said instructions during operation. Processor 210 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. Memory 220 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0100] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0101] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data verification method, comprising: Obtain multiple tables to be verified and parameter information of each table to be verified, wherein the parameter information of each table to be verified includes the data volume of the table to be verified, attribute information for identifying whether the table to be verified has a primary key, and identification information for identifying whether the table to be verified needs to be sharded. The multiple tables to be verified are sorted according to the data volume and attribute information of each table to be verified in order to generate a priority queue; The table of items to be verified that need to be sharded in the priority queue is obtained based on the identification information; The table to be verified that needs to be sharded is sharded, and the sharded table to be verified is moved to the tail of the priority queue to generate the sequential queue to be verified. Perform a data verification operation on the sequential queue to be verified.

2. The data verification method according to claim 1, wherein, The step of sorting the multiple tables to be verified according to the data volume and attribute information of each table to be verified includes: Based on the data volume and attribute information of each of the tables to be verified, the priority of each table to be verified is marked; The multiple tables to be verified are sorted according to their respective priorities.

3. The data verification method according to claim 2, wherein, The step of marking the priority of each table to be verified based on the data volume and attribute information of each table to be verified includes: The data volume of the table to be verified is compared with a preset data volume threshold, and the priority of the table to be verified is adjusted according to the comparison result; and Determine whether the table to be verified has a primary key based on the attribute information; If not, then lower the priority of the table to be verified.

4. The data verification method according to claim 1, wherein, The steps of sharding the table to be verified that requires sharding, and moving the sharded table to the tail of the priority queue to generate the sequential queue to be verified include: The table to be verified that needs to be sharded is sharded using a preset sharding algorithm to obtain multiple smaller tables; Remove the table to be verified that needs to be sharded from the priority queue; Multiple smaller tables are sequentially placed at the tail of the priority queue, and each smaller table is used as a table to be verified, thereby generating the sequential queue to be verified.

5. The data verification method according to claim 4, wherein, The steps of partitioning the table to be verified using a preset partitioning algorithm to obtain multiple smaller tables include: Obtain a preset sharding benchmark corresponding to the table to be sharded; Based on the sharding criteria, the table to be verified that needs to be sharded is sharded to obtain multiple smaller tables.

6. The data verification method according to claim 1, wherein, The steps for performing data verification on the sequential queue to be verified include: According to the preset number of parallel operations, the same number of tables to be verified are sequentially taken from the sequential queue to be verified. Parallel comparisons are performed on the same number of tables to be verified as the parallel number in each group.

7. The data verification method according to claim 1, wherein, After the step of performing data verification operation on the sequential queue to be verified, the data verification method further includes: The verification results are stored in the metadata database.

8. A machine-readable storage medium having a machine-executable program stored thereon, the machine-executable program, when executed by a processor, implementing the data verification method according to any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements the data verification method according to any one of claims 1 to 7.

Images