Data table verification method, device, equipment, storage medium and program

Abstract

The present application provides a data table verification method, device, equipment, storage medium and program, which are applied to the field of data processing technology. The method includes: obtaining a data table to be verified and a reference data table with the same table structure; processing the data table to be verified to obtain a first virtual data table including at least one first record, and the first number of occurrences of each first record in the data table to be verified; processing the reference data table to obtain a second virtual data table including at least one second record, and the second number of occurrences of each second record in the reference data table; according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table, determine whether the data table to be verified and the reference data table are consistent. The above consistency verification process does not rely on the primary key and can be applied to data table verification scenarios without primary keys.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data table verification method, device, equipment, storage medium and program. Background Art

[0002] In some scenarios, it is necessary to perform consistency check on the data in two data tables. For example, in a data migration scenario, after migrating the source database to the target database, it is necessary to perform consistency check on the data tables in the source database and the data tables in the target database to ensure the accuracy of the data migration.

[0003] Generally, when performing consistency check on two data tables, it is necessary to compare the two data tables based on the primary key (also called unique key). Taking data table a and data table b as examples, both data tables have primary keys. The records corresponding to the same primary key in data table a and data table b are compared to determine whether the data in the two data tables are consistent.

[0004] However, in actual business scenarios, some data tables do not have primary keys. How to perform consistency check on two data tables without primary keys is a technical problem that needs to be solved. Summary of the invention

[0005] The embodiments of the present application provide a data table verification method, apparatus, device, storage medium and program, which are used to provide a verification method for a data table without a primary key.

[0006] In a first aspect, an embodiment of the present application provides a data table verification method, comprising:

[0007] Obtaining a data table to be verified and a reference data table, wherein the data table to be verified and the reference data table have the same table structure;

[0008] Processing the data table to be verified to obtain a first virtual data table including at least one first record, and determining the first occurrence number of each first record in the first virtual data table in the data table to be verified; wherein each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified;

[0009] Processing the reference data table to obtain a second virtual data table including at least one second record, and determining the second occurrence number of each second record in the second virtual data table in the reference data table; wherein each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table;

[0010] Determine whether the data table to be checked is consistent with the reference data table based on the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table.

[0011] In a possible implementation, determining whether the data table to be verified is consistent with the reference data table according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table includes:

[0012] Generate a first difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the first difference record set includes: M records that are redundant in the to-be-checked data table compared to the reference data table; M is an integer greater than or equal to 0;

[0013] Generate a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the second difference record set includes: N records missing from the to-be-checked data table compared to the reference data table; N is an integer greater than or equal to 0;

[0014] According to the first difference record set and the second difference record set, it is determined whether the to-be-checked data table is consistent with the reference data table.

[0015] In a possible implementation, generating a first difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table includes:

[0016] Initialize the first difference record set to be empty;

[0017] For each first record in the first virtual data table, perform the following steps respectively:

[0018] Determining in the second virtual data table whether there is a target second record matching the first record;

[0019] If so, determining a difference K1 between a first number of occurrences corresponding to the first record and a second number of occurrences corresponding to the target second record;

[0020] If K1 is greater than 0, K1 first records are added to the first difference record set.

[0021] In a possible implementation, generating a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table includes:

[0022] Initialize the second difference record set to be empty;

[0023] For each second record in the second virtual data table, perform the following steps respectively:

[0024] Determining in the first virtual data table whether there is a target first record matching the second record;

[0025] If so, determining a difference K2 between a second number of occurrences corresponding to the second record and a first number of occurrences corresponding to the target first record;

[0026] If the difference K2 is greater than 0, K2 second records are added to the second difference record set.

[0027] In a possible implementation, determining whether the to-be-checked data table is consistent with the reference data table according to the first difference record set and the second difference record set includes:

[0028] If the first difference record set is not empty, and / or the second difference record set is not empty, it is determined that the to-be-checked data table is inconsistent with the reference data table; or;

[0029] If the first difference record set is empty, and the second difference record set is empty, it is determined that the to-be-checked data table is consistent with the reference data table.

[0030] In a possible implementation, the method further includes:

[0031] If it is determined that the data table to be checked is inconsistent with the reference data table, the data table to be checked is repaired according to the first difference record set and the second difference record set to obtain a repaired data table, and the repaired data table is consistent with the reference data table.

[0032] In a possible implementation, repairing the to-be-verified data table according to the first difference record set and the second difference record set to obtain a repaired data table includes:

[0033] If the first difference record set is not empty, deleting the records in the first difference record set from the to-be-checked data table; and / or,

[0034] If the second difference record set is not empty, then the records in the second difference record set are added to the to-be-checked data table.

[0035] In a possible implementation, the to-be-verified data table is a data table in a first database, and the reference data table is a data table in a second database;

[0036] The first database is a target database corresponding to a preset data migration task, and the second database is a source database corresponding to the preset data migration task; or

[0037] The first database is a standby database, and the second database is a primary database.

[0038] In a second aspect, an embodiment of the present application provides a data table verification device, including:

[0039] An acquisition module, used for acquiring a to-be-checked data table and a reference data table, wherein the to-be-checked data table and the reference data table have the same table structure;

[0040] A first processing module is used to process the data table to be verified to obtain a first virtual data table including at least one first record, and determine the first occurrence number of each first record in the first virtual data table in the data table to be verified; wherein each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified;

[0041] A second processing module is used to process the reference data table to obtain a second virtual data table including at least one second record, and determine the second occurrence number of each second record in the second virtual data table in the reference data table; wherein each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table;

[0042] A determination module is used to determine whether the data table to be checked is consistent with the reference data table based on the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores a computer program, and the computer program is configured to be executed by the processor to implement a method as described in any one of the first aspects.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the first aspects.

[0046] The data table verification method, device, equipment, storage medium and program provided by the embodiments of the present application include: obtaining a data table to be verified and a reference data table, wherein the table structures corresponding to the data table to be verified and the reference data table are the same; processing the data table to be verified to obtain a first virtual data table including at least one first record, and determining the first number of occurrences of each first record in the first virtual data table in the data table to be verified; wherein the first records in the first virtual data table are different from each other, and each first record appears at least once in the data table to be verified; processing the reference data table to obtain a second virtual data table including at least one second record, and determining the second number of occurrences of each second record in the second virtual data table in the reference data table; wherein the second records in the second virtual data table are different from each other, and each second record appears at least once in the reference data table; determining whether the data table to be verified and the reference data table are consistent according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table. In the above process, by expressing the data table to be verified in the form of "the first virtual data table, and the first number of occurrences corresponding to each first record in the first virtual data table", and expressing the reference data table in the form of "the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table", since there are no duplicate records in the first virtual data table and the second virtual data table, the consistency check of the data table to be verified and the reference data table can be conveniently and accurately implemented according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table. The above consistency check process does not rely on the primary key and can be applied to the scenario of checking data tables without primary keys. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a possible application scenario provided by an embodiment of the present application;

[0048] Figure 2 A schematic diagram of another possible application scenario provided for an embodiment of the present application;

[0049] Figure 3 A schematic diagram of another possible application scenario provided for an embodiment of the present application;

[0050] Figure 4 A flowchart of a data table verification method provided in an embodiment of the present application;

[0051] Figure 5A A schematic diagram of a data table to be verified and its processing results provided in an embodiment of the present application;

[0052] Figure 5B A schematic diagram of a reference data table and processing results provided for an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a flow chart of another data table verification method provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a data table verification process provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of a repair process of a data table to be verified provided in an embodiment of the present application;

[0056] Fig. 9 A flowchart of another data table verification method provided in an embodiment of the present application;

[0057] Fig.10 A schematic diagram of the structure of a data table verification device provided in an embodiment of the present application;

[0058] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The terms "first", "second", etc. in the specification, claims, and the above-mentioned drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0061] It should be understood that the terms "including" and "having" and any variations thereof used herein are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0062] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0063] First, the concepts and terms involved in the embodiments of the present application are explained.

[0064] A database is a warehouse that organizes, stores and manages data according to a certain data structure. The essence of a database is a file system. Users can add, delete, modify and query data in the database.

[0065] The database stores data in tables as organizational units. These tables are called data tables. A database can contain multiple data tables. Each data table includes a header and table content. The header is used to define the table structure. For example, the data table includes several fields, the field name, field type, field width, etc. of each field. Usually, a column in a data table corresponds to a field. The table content usually includes multiple rows, and the data in each row can be called a record.

[0066] Typically, there is a combination of one or more columns in a data table whose values ​​can uniquely identify each row in the table. Such a column or columns are called the primary key of the data table.

[0067] In order to facilitate the understanding of the technical solution of this application, Figure 1 and Figure 2 Several possible application scenarios of the embodiments of the present application are introduced.

[0068] Figure 1 A schematic diagram of a possible application scenario provided by an embodiment of the present application. The application scenario is a data migration scenario between databases. Figure 1As shown, the data in database A can be migrated to database B. In this application scenario, database A can be called the source database, and database B can be called the target database. In the data migration process, there are inevitably some factors that cause the data migration process to be abnormal, making the data in database B inconsistent with that in database A. Therefore, after the data migration is completed, the data table verification device can read the data table from database B and read the data table from database A, and then verify the data tables in the two databases to determine whether the data in the two databases are consistent.

[0069] In one example, database A is a local database and database B is a cloud database, that is, the data stored in the local database is migrated to the cloud database. Alternatively, database A is a first cloud database and database B is a second cloud database. For example, data stored in a public cloud database is migrated to a private cloud database, or data stored in a private cloud database is migrated to a public cloud database.

[0070] In another example, database A is used for online transaction processing (OLTP) and database B is used for online analytical processing (OLAP). OLTP is mainly used to record the occurrence of certain business events, such as purchase behavior. When the behavior occurs, the system will record who did what, when and where. Such a row (or multiple rows) of data will be updated in database A by adding, deleting and modifying. It requires high real-time performance, strong stability and ensuring that the data is updated successfully in time. When the data in database A accumulates to a certain extent and it is necessary to make a summary analysis of what happened in the past, it is necessary to migrate the data in database A to database B and perform OLAP in database B.

[0071] Figure 2 A schematic diagram of another possible application scenario provided by an embodiment of the present application. The application scenario is a database master-slave synchronization scenario. Figure 2As shown, after the business system generates business data, the business data is written into database A, and database A synchronizes the business data to database B periodically or in real time. In this application scenario, database A can be called the master database, and database B can be called the slave database or the backup database. Optionally, database A and database B can be deployed in different geographical areas, respectively. In this application scenario, when an exception or failure occurs in the data synchronization process, the data in database B may be inconsistent with that in database A. Therefore, the data table verification device can read the data table from database B and read the data table from database A, and then verify the data tables in the two databases. The above verification process can be performed in real time or periodically to promptly detect inconsistencies between the two databases.

[0072] Figure 3 A schematic diagram of another possible application scenario provided by the embodiment of the present application. The application scenario is a database multi-write scenario. Figure 3 As shown, after the business system generates business data, the business data is written into database A and database B respectively. Ideally, the data in database A and database B are consistent. However, in actual applications, there may be some abnormal situations that make the data written in database A and database B inconsistent. Therefore, the data table verification device can read the data table from database B and read the data table from database A, and then verify the data tables in the two databases. The above verification process can be performed in real time or periodically to promptly detect the inconsistency between the two databases.

[0073] based on Figures 1 to 3 In the application scenario shown, there may be one or more data tables in database A and database B, and only one data table is used as an example in the attached figure. When verifying database A and database B, it is necessary to verify the data tables in database A and the data tables in database B respectively. For example, assuming that database A includes data tables a1 and a2, and database B includes data tables b1 and b2, when verifying database A and database B, it is necessary to perform consistency verification on data tables a1 and b1, and consistency verification on data tables a2 and b2.

[0074] Generally, when performing consistency check on two data tables, it is necessary to compare the two data tables based on the primary key (also called unique key). Taking data table a1 and data table b1 as an example, both data tables have primary keys. The records corresponding to the same primary key in data table a1 and data table b1 are compared to determine whether the data in the two data tables are consistent.

[0075] However, in actual business scenarios, some data tables do not have primary keys. For two data tables without primary keys, the above verification method is no longer applicable.

[0076] To this end, the embodiment of the present application provides a data table verification method, device, equipment, storage medium and program, which can be applied to verify a data table without a primary key. In the embodiment of the present application, considering that there may be duplicate records in the data table without a primary key, if the verification is performed directly by comparing the records one by one, the accuracy of the verification result cannot be guaranteed. Therefore, in the embodiment of the present application, before the data table is verified, the data table to be verified is first expressed as a "first virtual data table composed of mutually different first records that have appeared in the data table to be verified, and the first number of occurrences corresponding to each first record in the first virtual data table", and the reference data table is expressed as a "second virtual data table composed of mutually different second records that have appeared in the reference data table, and the second number of occurrences corresponding to each second record in the second virtual data table". Since there are no duplicate records in the first virtual data table and the second virtual data table, according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table, the consistency check of the data table to be verified and the reference data table can be conveniently and accurately realized.

[0077] The technical solutions provided by the embodiments of the present application are described in detail below through specific embodiments. It should be noted that the technical solutions provided by the embodiments of the present application may include part or all of the following contents, and the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0078] Figure 4 The flowchart of a data table verification method provided in an embodiment of the present application is shown in FIG. The method of this embodiment can be executed by a data table verification device. Figure 4 As shown, the method of this embodiment includes:

[0079] S401: Acquire a data table to be verified and a reference data table, wherein the data table to be verified and the reference data table have the same table structure.

[0080] In this embodiment, the data table to be verified is a data table in the first database, and the reference data table is a data table in the second database.

[0081] In one example, the first database is a target database corresponding to a preset data migration task, and the second database is a source database corresponding to the preset data migration task. Figure 1 In the application scenario shown, the data table to be verified may be a data table in database B (target database), and the reference data table may be a data table in database A (source database).

[0082] In another example, the first database is the standby database and the second database is the primary database. Figure 2 In the application scenario, the data table to be verified may be a data table in database B (standby database / slave database), and the reference data table may be a data table in database A (master database).

[0083] In another example, combining Figure 3 In the application scenario shown, the data table to be verified may be database A, and the reference data table may be database B; alternatively, the data table to be verified may be database B, and the reference data table may be database A.

[0084] In the embodiment of the present application, the table structures corresponding to the data table to be checked and the reference data table are the same, which means that the fields included in the header of the data table to be checked are the same as the fields included in the header of the reference data table. For example, the header of the data table to be checked includes field 1 and field 2, and the header of the reference data table also includes field 1 and field 2. In addition, the type, width and other information of field 1 in the data table to be checked are the same as the type, width and other information of field 1 in the reference data table, and the type, width and other information of field 2 in the data table to be checked are the same as the type, width and other information of field 2 in the reference data table. The data table to be checked and the reference data table can also be called isomorphic data tables.

[0085] The method of this embodiment can be executed by an electronic device, in which a data table verification device is deployed. The electronic device can access the data table to be verified and the reference data table, and read the data in the data table to be verified into the memory, and read the data in the reference data table into the memory, so as to execute the subsequent steps of this embodiment.

[0086] In a possible implementation, the database to be verified is a data table in a first database, and the reference database is a data table in a second database. The first database is deployed locally, and the second database is deployed remotely. A database link channel (e.g., dblink) is provided between the first database and the second database. The electronic device can directly access the first database and obtain the data table to be verified from the first database. Furthermore, the electronic device can access the second database through the database link channel and obtain the reference data table from the second database.

[0087] In this implementation, a database link channel is deployed between the first database and the second database, and the data table is read through the database link channel, so that the computing power of the database can be used to improve the reading speed of the data table.

[0088] In another possible implementation, the electronic device is deployed with a big data computing engine (such as spark), and can access the first database through the big data computing engine and obtain the data table to be verified from the first database; and access the second database through the big data computing engine and obtain the reference data table from the second database.

[0089] In this implementation, the data table is obtained from the database through the big data computing engine, which can simplify the data table reading process on the one hand and improve the data table reading speed on the other hand.

[0090] S402: Process the data table to be verified to obtain a first virtual data table including at least one first record, and determine the first number of occurrences of each first record in the data table to be verified in the first virtual data table; wherein the first records in the first virtual data table are different from each other, and each first record appears at least once in the data table to be verified.

[0091] S403: Process the reference data table to obtain a second virtual data table including at least one second record, and determine the second occurrence number of each second record in the second virtual data table in the reference data table; wherein, each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table.

[0092] In this embodiment, a row of data in a data table is called a record. Assuming that the data table includes P columns, that is, the data table corresponds to P fields, a record includes the values ​​of P fields. In a data table, if there are records X and Y, and the values ​​of each field in record X are the same as the values ​​of the corresponding fields in record Y, then record X and record Y are called duplicate records.

[0093] In a scenario without a primary key, duplicate records may exist in both the data table to be verified and the reference data table. In the case of duplicate records, if the data table to be verified and the reference data table are verified directly by matching records one by one, the verification result will be inaccurate. Therefore, in an embodiment of the present application, before the data table to be verified and the reference data table are verified, the data table to be verified is first processed to obtain a first virtual data table composed of mutually different first records that have appeared in the data table to be verified, and the first number of occurrences of each first record in the first virtual data table in the data table to be verified is determined; and the reference data table is processed to obtain a second virtual data table composed of mutually different second records that appear in the reference data table, and the second number of occurrences of each second record in the second virtual data table in the reference data table is determined.

[0094] Among them, the above-mentioned first virtual data table can be understood as the data table obtained after de-duplicating the records in the data table to be verified (that is, only one of the duplicate records is retained). That is to say, each record that appears in the data table to be verified is located in the above-mentioned first virtual data table, and the first records in the above-mentioned first virtual data table are different from each other. For each first record in the above-mentioned first virtual data table, the first occurrence times of the first record in the data table to be verified can also be counted. In this way, using the above-mentioned first virtual data table and the first occurrence times corresponding to the first records in the first virtual data table, the data in the data table to be verified can be completely expressed.

[0095] The above-mentioned second virtual data table can be understood as the data table obtained after de-duplicating the records in the reference data table (that is, only one of the duplicate records is retained). That is to say, each record that appears in the reference data table is located in the above-mentioned second virtual data table, and the second records in the above-mentioned second virtual data table are different from each other. For each second record in the above-mentioned second virtual data table, the second occurrence times of the second record in the reference data table can also be counted. In this way, using the above-mentioned second virtual data table and the second occurrence times corresponding to the second records in the second virtual data table, the data in the reference data table can be completely expressed.

[0096] It should be noted that the embodiments of the present application do not limit the execution order of S402 and S403. S402 can be executed first and then S403, or S403 can be executed first and then S402, and the two can also be executed in parallel.

[0097] The following Figure 5A and Figure 5B are used for illustration by way of example.

[0098] As an example, Figure 5A is a schematic diagram of the data table to be verified and its processing results provided by the embodiments of the present application. As Figure 5A shown, it is assumed that the data table to be verified includes 2 fields, namely field 1 and field 2. The data table to be verified includes 4 records, namely {A, Jia}, {A, Jia}, {A, Yi}, {B, Yi}. Among them, {A, Jia} means that the value of field 1 is Jia and the value of field 2 is A.

[0099] Exemplarily, the data table to be verified can be processed as follows to obtain the first virtual data table and the first occurrence times corresponding to the first records in the first virtual data table:

[0100] (1) Initialize the first virtual data table to be empty.

[0101] (2) Add the first record {A, Jia} in the data table to be verified to the first virtual data table, and record the first occurrence count corresponding to the record {A, Jia} as 1.

[0102] (3) Match the second record {A, Jia} in the data table to be verified with the records in the first virtual data table. Since the second record {A, Jia} matches the record {A, Jia} in the first virtual data table, the second record {A, Jia} is a duplicate record and does not need to be added to the first virtual data table. In this case, the first occurrence count corresponding to the record {A, Jia} in the first virtual data table needs to be incremented by 1.

[0103] (4) Match the third record {A, Yi} in the data table to be verified with the records in the first virtual data table. Since the third record {A, Yi} does not match the record {A, Jia} in the first virtual data table, add the third record {A, Yi} to the first virtual data table, and record the first occurrence count corresponding to the record {A, Yi} as 1.

[0104] (5) Match the fourth record {B, Yi} in the data table to be verified with the records in the first virtual data table. Since the third record {B, Yi} does not match the record {A, Jia} or the record {A, Yi} in the first virtual data table, add the fourth record {B, Yi} to the first virtual data table, and record the first occurrence count corresponding to the record {B, Yi} as 1.

[0105] Continue to refer to Figure 5A , after the above processing, the first virtual data table obtained includes the following 3 first records: {A, Jia}, {A, Yi}, {B, Yi}. Among them, the first occurrence count corresponding to {A, Jia} is 2, the first occurrence count corresponding to {A, Yi} is 1, and the first occurrence count corresponding to {B, Yi} is 1.

[0106] Figure 5B is a schematic diagram of the reference data table and its processing results provided by the embodiments of the present application. As Figure 5B shown, assume that the reference data table includes 5 records, namely {A, Jia}, {A, Yi}, {A, Yi}, {B, Yi}, {C, Yi}. Exemplarily, the reference data table can be processed as follows to obtain a second virtual data table and the second occurrence counts corresponding to each second record in the second virtual data table:

[0107] (1) Initialize the second virtual data table to be empty.

[0108] (2) Add the first record {A, Jia} in the reference data table to the second virtual data table, and record the second occurrence count corresponding to the record {A, Jia} as 1.

[0109] (3) Match the second record {B, A} in the reference data table with the records in the second virtual data table. Since the second record {B, A} does not match the record {A, A} in the second virtual data table, the second record {B, A} is added to the second virtual data table, and the second occurrence count corresponding to the record {B, A} is recorded as 1.

[0110] (4) Match the third record {B, A} in the reference data table with the records in the second virtual data table. Since the third record {B, A} matches the record {B, A} in the second virtual data table successfully, the third record {B, A} is a duplicate record and does not need to be added to the second virtual data table. In this case, the second occurrence count corresponding to the record {B, A} in the second virtual data table needs to be incremented by 1.

[0111] (5) Match the fourth record {B, B} in the reference data table with the records in the second virtual data table. Since the fourth record {B, B} does not match the record {A, A} and the record {B, A} in the second virtual data table, the fourth record {B, B} is added to the second virtual data table, and the second occurrence count corresponding to the record {B, B} is recorded as 1.

[0112] (6) Match the fifth record {B, C} in the reference data table with the records in the second virtual data table. Since the fifth record {B, C} does not match the record {A, A}, the record {B, A}, and the record {B, B} in the second virtual data table, the fifth record {B, C} is added to the second virtual data table, and the second occurrence count corresponding to the record {B, C} is recorded as 1.

[0113] Continue to refer to Figure 5B After the above processing, the obtained second virtual data table includes the following 4 second records: {A, A}, {B, A}, {B, B}, {B, C}. Among them, the second occurrence count corresponding to {A, A} is 1, the second occurrence count corresponding to {B, A} is 2, the second occurrence count corresponding to {B, B} is 1, and the second occurrence count corresponding to {B, C} is 1.

[0114] S404: Determine whether the data table to be verified is consistent with the reference data table according to the first virtual data table, the first occurrence counts corresponding to the first records in the first virtual data table, the second virtual data table, and the second occurrence counts corresponding to the second records in the second virtual data table.

[0115] In this embodiment, the consistency between the data table to be verified and the reference data table means that all records in the data table to be verified and the reference data table are completely consistent. The inconsistency between the data table to be verified and the reference data table means that at least some records in the data table to be verified and the reference data table are inconsistent. For example, record 1 exists in the data table to be verified, but record 1 does not exist in the reference data table. For another example, record 2 does not exist in the data table to be verified, but record 2 exists in the reference data table. For another example, record 3 exists in both the data table to be verified and the reference data table, but the number of occurrences of record 3 in the data table to be verified is inconsistent with the number of occurrences of record 3 in the reference data table.

[0116] In this embodiment, by expressing the data table to be verified in the form of "the first virtual data table, and the first number of occurrences corresponding to each first record in the first virtual data table", and expressing the reference data table in the form of "the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table", since there are no duplicate records in the first virtual data table and the second virtual data table, the consistency check of the data table to be verified and the reference data table can be conveniently and accurately implemented according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table.

[0117] For example, if record X exists in the first virtual data table, but record X does not exist in the second virtual data table, it is determined that the data table to be verified is inconsistent with the reference data table. For another example, if record Y does not exist in the first virtual data table, but record Y exists in the second virtual data table, it is determined that the data table to be verified is inconsistent with the reference data table. For another example, if record Z exists in the first virtual data table, record Z exists in the second virtual data table, and the first number of occurrences of record Z in the verification data table is different from the second number of occurrences in the reference data table, it is determined that the data table to be verified is inconsistent with the reference data table. For another example, if each first record in the second virtual data table matches each second record in the second virtual data table one by one, and the first number of occurrences of each first record in the verification data table is the same as the second number of occurrences of the second record matched with it in the reference data table, it is determined that the data table to be verified is consistent with the reference data table.

[0118] The data table verification method provided by the present embodiment includes: obtaining a data table to be verified and a reference data table, wherein the table structures corresponding to the data table to be verified and the reference data table are the same; processing the data table to be verified to obtain a first virtual data table including at least one first record, and determining the first number of occurrences of each first record in the first virtual data table in the data table to be verified; wherein the first records in the first virtual data table are different from each other, and each first record appears at least once in the data table to be verified; processing the reference data table to obtain a second virtual data table including at least one second record, and determining the second number of occurrences of each second record in the second virtual data table in the reference data table; wherein the second records in the second virtual data table are different from each other, and each second record appears at least once in the reference data table; and determining whether the data table to be verified and the reference data table are consistent according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table. In the above process, by expressing the data table to be verified in the form of "the first virtual data table, and the first number of occurrences corresponding to each first record in the first virtual data table", and expressing the reference data table in the form of "the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table", since there are no duplicate records in the first virtual data table and the second virtual data table, the consistency check of the data table to be verified and the reference data table can be conveniently and accurately implemented according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table. The above consistency check process does not rely on the primary key and can be applied to the scenario of checking data tables without primary keys.

[0119] exist Figure 4 Based on the embodiment shown, the following Figures 6 to 9 Several specific embodiments illustrate the technical solution of the present application in more detail.

[0120] Figure 6 A schematic diagram of another data table verification method provided in an embodiment of the present application. Figure 6 As shown, the method of this embodiment includes:

[0121] S601: Acquire a data table to be verified and a reference data table, wherein the data table to be verified and the reference data table have the same table structure.

[0122] S602: Process the data table to be verified to obtain a first virtual data table including at least one first record, and determine the first number of occurrences of each first record in the data table to be verified in the first virtual data table; wherein, each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified.

[0123] S603: Process the reference data table to obtain a second virtual data table including at least one second record, and determine the second occurrence number of each second record in the second virtual data table in the reference data table; wherein, each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table.

[0124] It should be understood that the specific implementation of the above S601 to S603 is different from Figure 4 The embodiments shown are similar and will not be described in detail here.

[0125] S604: Generate a first difference record set based on the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the first difference record set includes: M records in excess of the data table to be verified compared to the reference data table; M is an integer greater than or equal to 0.

[0126] In this embodiment, by comparing the first number of occurrences corresponding to each first record in the first virtual data table and the first virtual data table with the second number of occurrences corresponding to each second record in the second virtual data table and the second virtual data table, it is possible to determine which records are redundant with the reference data table in the data table to be verified. These redundant records constitute a first difference record set. The first difference record set includes M records. M can be equal to 0 or greater than 0. It should be understood that when M=0, it means that the data table to be verified does not have any redundant records with the reference data table.

[0127] In one possible implementation, the first difference record set can be generated in the following manner: initialize the first difference record set to empty; for each first record in the first virtual data table, perform the following steps respectively: determine whether there is a target second record matching the first record in the second virtual data table; if so, determine the difference K1 between the first number of occurrences corresponding to the first record and the second number of occurrences corresponding to the target second record; if K1 is greater than 0, add K1 of the first records to the first difference record set.

[0128] The first record matches the target second record, which means that the values ​​of each field in the first record are exactly the same as the values ​​of each field in the target second record. Exemplarily, a database query statement (such as a select statement) can be used to query the first record in the second virtual data table. If the returned query result is empty, it means that there is no target second record matching the first record in the second virtual data table; if the returned query result is not empty, the target second record can be determined in the second virtual data table according to the returned query result.

[0129] S605: Generate a second difference record set based on the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the second difference record set includes: N records missing from the data table to be verified compared to the reference data table; N is an integer greater than or equal to 0.

[0130] In this embodiment, by comparing the first number of occurrences corresponding to each first record in the first virtual data table and the first virtual data table with the second number of occurrences corresponding to each second record in the second virtual data table and the second virtual data table, it is possible to determine which records are missing from the data table to be verified compared to the reference data table. These missing records constitute a second difference record set. The second difference record set includes N records. N can be equal to 0 or greater than 0. It should be understood that when N=0, it means that the data table to be verified does not lack any records compared to the reference data table.

[0131] In one possible implementation, the second difference record set can be generated in the following manner: initialize the second difference record set to empty; for each second record in the second virtual data table, perform the following steps respectively: determine whether there is a target first record matching the second record in the first virtual data table; if so, determine the difference K2 between the second number of occurrences corresponding to the second record and the first number of occurrences corresponding to the target first record; if K2 is greater than 0, add K1 of the second records to the second difference record set.

[0132] The second record matches the target first record, which means that the values ​​of each field in the second record are exactly the same as the values ​​of each field in the target first record. Exemplarily, a database query statement (such as a select statement) can be used to query the second record in the first virtual data table. If the returned query result is empty, it means that there is no target first record matching the second record in the first virtual data table; if the returned query result is not empty, the target first record can be determined in the first virtual data table according to the returned query result.

[0133] It should be noted that the execution order of S604 and S605 in this embodiment is not limited. S604 can be executed first and then S605, or S605 can be executed first and then S604, and the two can also be executed in parallel.

[0134] The following Figure 7 is an example to illustrate S604 and S605.

[0135] Figure 7 is a schematic diagram of the data table verification process provided by the embodiment of the present application. This embodiment Figure 5A and Figure 5B is an example based on the data table to be verified, the reference data table and their processing results shown.

[0136] Refer to Figure 7 , the first virtual data table includes 3 first records, namely {A, Jia}, {A, Yi}, {B, Yi}. The second virtual data table includes 4 second records, namely {A, Jia}, {A, Yi}, {B, Yi}, {C, Yi}.

[0137] The generation process of the first difference record set is as follows:

[0138] (1) First, initialize the first difference record set to be empty.

[0139] (2) For the first record {A, Jia}, match this first record {A, Jia} with each second record in the second virtual data table to determine the target second record {A, Jia}. The first occurrence count corresponding to the first record {A, Jia} is 2, and the second occurrence count corresponding to the target second record {A, Jia} is 1. The difference between the two is K1 = 2 - 1 = 1, indicating that there is 1 more record {A, Jia} in the data table to be verified than in the reference data table. Therefore, add 1 record {A, Jia} to the first difference record set.

[0140] (3) For the first record {A, Yi}, match this first record {A, Yi} with each second record in the second virtual data table to determine the target second record {A, Yi}. The first occurrence count corresponding to the first record {A, Yi} is 1, and the second occurrence count corresponding to the target second record {A, Yi} is 2. The difference between the two is K1 = 1 - 2 = -1, indicating that there is no extra record {A, Yi} in the data table to be verified compared to the reference data table. In this case, there is no need to add the record {A, Yi} to the first difference record set.

[0141] (4) For the first record {B, B}, match this first record {A, B} with each second record in the second virtual data table to determine the target second record {B, B}. The first occurrence count corresponding to the first record {B, B} is 1, and the second occurrence count corresponding to the target second record {B, B} is 1. The difference between the two is K1 = 1 - 1 = 0, indicating that the data table to be verified does not have any redundant record {B, B} compared to the reference data table. In this case, there is no need to add the record {B, B} to the first set of differential records.

[0142] After the above comparison process, the first set of differential records generated includes: {A, A}. That is to say, the data table to be verified has a redundant record {A, A} compared to the reference data table.

[0143] The generation process of the second set of differential records is as follows:

[0144] (1) First, initialize the second set of differential records to be empty.

[0145] (2) For the second record {A, A}, match this second record {A, A} with each first record in the first virtual data table to determine the target first record {A, A}. The second occurrence count corresponding to the second record {A, A} is 1, and the first occurrence count corresponding to the target first record {A, A} is 2. The difference between the two is K2 = 1 - 2 = -1, indicating that the data table to be verified does not have any missing record {A, A} compared to the reference data table. In this case, there is no need to add the record {A, A} to the second set of differential records.

[0146] (3) For the second record {A, B}, match this second record {A, B} with each first record in the first virtual data table to determine the target first record {A, B}. The second occurrence count corresponding to the second record {A, B} is 2, and the first occurrence count corresponding to the target first record {A, B} is 1. The difference between the two is K2 = 2 - 1 = 1, indicating that the data table to be verified is missing 1 record {A, B} compared to the reference data table. Therefore, add 1 record {A, B} to the second set of differential records.

[0147] (4) For the second record {B, B}, match this second record {B, B} with each first record in the first virtual data table to determine the target first record {B, B}. The second occurrence count corresponding to the second record {B, B} is 1, and the first occurrence count corresponding to the target first record {B, B} is 1. The difference between the two is K2 = 1 - 1 = 0, indicating that the data table to be verified does not have any missing record {B, B} compared to the reference data table. In this case, there is no need to add the record {B, B} to the second set of differential records.

[0148] (5) For the second record {B, C}, the second record {B, C} is matched with each first record in the first virtual data table. It is found that there is no target first record matching it in the first virtual data table, indicating that there is no record {B, C} in the data table to be verified. The second occurrence count corresponding to the second record {B, C} is 1, indicating that the data table to be verified lacks one record {B, C} compared to the reference data table. Therefore, one record {B, C} is added to the second difference record set.

[0149] After the above comparison process, the generated second difference records include: {B, A} and {B, C}. That is, the to-be-checked data table lacks the records {B, A} and {B, C} compared to the reference data table.

[0150] S606: Determine whether the to-be-checked data table is consistent with the reference data table according to the first difference record set and the second difference record set.

[0151] Exemplarily, if the first difference record set is empty and the second difference record set is empty, it means that the data table to be verified has neither redundant records nor missing records compared with the reference data table. Therefore, in this case, it is determined that the data table to be verified is consistent with the reference data table.

[0152] If the first difference record set is not empty and the second difference record set is empty, it means that the data table to be checked has redundant records compared with the reference data table. Therefore, in this case, it is determined that the data table to be checked is inconsistent with the reference data table.

[0153] If the first difference record set is empty and the second difference record set is not empty, it means that the data table to be checked has missing records compared with the reference data table. Therefore, in this case, it is determined that the data table to be checked is inconsistent with the reference data table.

[0154] If the first difference record set is not empty and the second difference record set is not empty, it means that the data table to be verified has both redundant records and missing records compared with the reference data table. Therefore, in this case, it is determined that the data table to be verified is inconsistent with the reference data table.

[0155] Based on the above, Figure 7 In the example shown, since the first difference record set is not empty and the second difference record set is not empty, it is determined that the data table to be verified is inconsistent with the reference data table.

[0156] In this embodiment, by generating a first difference record set and a second difference record set based on the first virtual data table, the first occurrence times corresponding to the first records in the first virtual data table, the second virtual data table, and the second occurrence times corresponding to the second records in the second virtual data table, when performing data table verification, both the redundant records in the data table to be verified compared with the reference data table and the missing records in the data table to be verified compared with the reference data table are considered, improving the accuracy of the data table verification result.

[0157] In some possible implementation manners, after S606, the following S607 may also be continuously executed.

[0158] S607: If it is determined that the data table to be verified is inconsistent with the reference data table, then repair the data table to be verified according to the first difference record set and the second difference record set to obtain a repaired data table, and the repaired data table is consistent with the reference data table.

[0159] That is to say, when it is determined that the data table to be verified is inconsistent with the reference data table, the data table to be verified can also be repaired to ensure the consistency of the two databases.

[0160] Exemplarily, the manner of repairing the data table to be verified may include one or both of the following:

[0161] (1) If the first difference record set is not empty, then delete the records in the first difference record set from the data table to be verified. That is to say, delete the "redundant records in the data table to be verified compared with the reference data table" from the data table to be verified.

[0162] (2) If the second difference record set is not empty, then add the records in the second difference record set to the data table to be verified. That is to say, add the "missing records in the data table to be verified compared with the reference data table" to the data table to be verified.

[0163] Figure 8 It is a schematic diagram of the repair process of the data table to be verified provided by the embodiment of the present application. As Figure 8 shown, still taking the data table to be verified in Figure 5A as an example for illustration. Suppose the first difference record set includes: {A, B}, then delete the record {A, B} from the data table to be verified to obtain an intermediate result data table. Suppose the second difference record set includes: {C, D} and {C, E}, then add the records {C, D} and {C, E} to the data table to be verified to obtain a repaired data table. It can be seen that the repaired data table is consistent with the reference data table.

[0164] In this embodiment, when it is determined that the data table to be checked is inconsistent with the reference data table, the data table to be checked can be repaired in real time to make the repaired data table consistent with the reference data table, thereby improving the real-time performance of the verification+repair.

[0165] On the basis of any of the above embodiments, when the amount of data in the data table to be verified and the reference data table is large, it takes a long time to perform consistency verification on the two. To this end, the embodiment of the present application can also perform sharding processing on the data table to be verified and the reference data table respectively, splitting the data table to be verified into multiple sharding tables to be verified, and splitting the reference data table into multiple reference sharding tables. Furthermore, multiple sharding tables to be verified and multiple reference sharding tables are verified in parallel using multithreading, thereby improving verification efficiency. Fig. 9 Provide detailed explanation.

[0166] Fig. 9 A flowchart of another data table verification method provided in an embodiment of the present application is shown below. Fig. 9 As shown, the data table verification method provided in this embodiment includes:

[0167] S901: Obtain a data table to be verified and a reference data table.

[0168] The table structures of the to-be-verified data table and the reference data table are the same. Figure 4 S401 in the description will not be described in detail here.

[0169] S902: Determine whether the to-be-verified data table and the reference data table support sharding processing.

[0170] If supported, continue to execute S903 to S906.

[0171] If not supported, continue to execute S907.

[0172] In this embodiment, since the table structures corresponding to the data table to be verified and the reference data table are the same, the results of whether the data table to be verified and the reference data table support sharding processing are consistent. That is, if the data table to be verified supports sharding processing, the reference data table also supports sharding processing. If the data table to be verified does not support sharding processing, the reference data table also does not support sharding processing.

[0173] In some possible implementations, whether the data table to be verified supports sharding can be determined based on whether the table contains index fields. If the table to be verified contains index fields, it means that the table supports sharding, otherwise it means that the table does not support sharding. The index field refers to a field that can be used to implement the search function, such as a time field, a type field, etc.

[0174] It should be understood that in addition to sharding according to index fields, other sharding schemes may also be used, and this embodiment does not specifically limit the sharding scheme.

[0175] S903: Slice the data table to be verified to obtain S sliced ​​tables to be verified.

[0176] S904: Slice the reference data table to obtain S slice tables to be verified.

[0177] Wherein, S is an integer greater than 1.

[0178] Exemplarily, after determining the index field, S non-overlapping value ranges can be obtained based on the value of the index field. For example, assuming that the index field is a time field, the following 24 value ranges can be divided: 0 o'clock to 1 o'clock, 1 o'clock to 2 o'clock, 2 o'clock to 3 o'clock, ..., 23 o'clock to 24 o'clock, and each value range corresponds to a sub-table.

[0179] For each record in the data table to be verified, determine the value range of the index field in the record, and add the record to the sub-table corresponding to the value range. In this way, the data table to be verified can be divided into 24 sub-tables to be verified.

[0180] For each record in the reference data table, determine the value range of the index field in the record, and add the record to the sub-table corresponding to the value range. In this way, the reference data table can be divided into 24 sub-tables to be checked.

[0181] S905: Verify and repair the S sub-tables to be verified and the S reference sub-tables respectively to obtain S repaired sub-tables.

[0182] It should be understood that since the data table to be checked and the reference data table adopt the same sub-table processing method, the S sub-tables to be checked and the S reference sub-tables are one-to-one corresponding. That is, the i-th sub-table to be checked corresponds to the i-th reference sub-table. The processing process of S905 is as follows: the i-th sub-table to be checked and the i-th reference sub-table are checked. If the check result is consistent, the i-th sub-table to be checked is used as the repaired i-th sub-table; if the check result is inconsistent, the i-th sub-table to be checked is modified to obtain the modified i-th sub-table. i is 1, 2, ..., S in sequence.

[0183] It should be understood that the verification and repair process for the i-th verification sub-table and the i-th reference sub-table can be referred to in Figure 6 The detailed description of the illustrated embodiment is not repeated here.

[0184] In this embodiment, S threads may be used to execute the checking and repairing process of the S sub-tables in parallel, thus improving the efficiency of checking and repairing the data tables.

[0185] S906: The repaired first sub-table, the second sub-table, ..., the Sth sub-table are merged to obtain a repaired data table.

[0186] The repaired data sheet is consistent with the data in the reference data sheet.

[0187] S907: Determine whether the total number of records in the data table to be verified is less than or equal to a preset threshold.

[0188] If yes, execute S908.

[0189] If not, then end.

[0190] S908: Verify the data table to be verified and the reference data table, and when the verification result is inconsistent, repair the data table to be verified to obtain a repaired data table.

[0191] In this embodiment, when the data table to be verified and the reference data table do not support sharding, if full table verification and repair are performed, the full table data of the data table to be verified and the reference data table need to be loaded into the memory. When the total number of records is large, there is a risk that the memory cannot hold the data. Therefore, in this embodiment of the application, the maximum number of records supported by the full table verification can be determined according to the memory situation, and the maximum number of records can be determined as the preset threshold.

[0192] If the total record data in the data table to be verified exceeds a preset threshold, a prompt message is output to the user to prompt the user that the current electronic device cannot support verification of the data table to be verified.

[0193] If the total number of records in the data table to be verified does not exceed the preset threshold, the data in the data table to be verified and the reference data table are loaded into the content, and the whole table is verified and repaired, so as to obtain a repaired data table. The repaired data table is consistent with the data in the reference data table. It should be understood that the implementation method of S908 is the same as Figure 6 The embodiments shown are similar and will not be described in detail here.

[0194] In this embodiment, by slicing the data table to be verified and the reference data table, the corresponding sub-tables in the data table to be verified and the reference data table can be verified and repaired respectively, and the verification and repair processes corresponding to different sub-tables can be executed in parallel, thereby improving the efficiency of data table verification and repair.

[0195] The above describes the data table verification method provided by the embodiment of the present application, and the following describes the data table verification device provided by the embodiment of the present application.

[0196] The embodiment of the present application can divide the functional modules of the data table verification device according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or software functional modules.

[0197] It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module using the corresponding functions.

[0198] Fig.10 This is a schematic diagram of the structure of a data table verification device provided in an embodiment of the present application. Fig.10 As shown, the data table verification device 1000 provided in this embodiment includes: an acquisition module 1001, a first processing module 1002, a second processing module 1003 and a determination module 1004. Among them,

[0199] The acquisition module 1001 is used to acquire a to-be-checked data table and a reference data table, wherein the to-be-checked data table and the reference data table have the same table structure;

[0200] A first processing module 1002 is used to process the data table to be verified to obtain a first virtual data table including at least one first record, and determine the first occurrence number of each first record in the first virtual data table in the data table to be verified; wherein each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified;

[0201] A second processing module 1003 is used to process the reference data table to obtain a second virtual data table including at least one second record, and determine the second occurrence number of each second record in the second virtual data table in the reference data table; wherein each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table;

[0202] Determination module 1004 is used to determine whether the data table to be checked is consistent with the reference data table based on the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table.

[0203] In a possible implementation, the determining module 1004 is specifically configured to:

[0204] Generate a first difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the first difference record set includes: M records that are redundant in the to-be-checked data table compared to the reference data table; M is an integer greater than or equal to 0;

[0205] Generate a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the second difference record set includes: N records missing from the to-be-checked data table compared to the reference data table; N is an integer greater than or equal to 0;

[0206] According to the first difference record set and the second difference record set, it is determined whether the to-be-checked data table is consistent with the reference data table.

[0207] In a possible implementation, the determining module 1004 is specifically configured to:

[0208] Initialize the first difference record set to be empty;

[0209] For each first record in the first virtual data table, perform the following steps respectively:

[0210] Determining in the second virtual data table whether there is a target second record matching the first record;

[0211] If so, determining a difference K1 between a first number of occurrences corresponding to the first record and a second number of occurrences corresponding to the target second record;

[0212] If K1 is greater than 0, K1 first records are added to the first difference record set.

[0213] In a possible implementation, the determining module 1004 is specifically configured to:

[0214] Initialize the second difference record set to be empty;

[0215] For each second record in the second virtual data table, perform the following steps respectively:

[0216] Determining in the first virtual data table whether there is a target first record matching the second record;

[0217] If so, determining a difference K2 between a second number of occurrences corresponding to the second record and a first number of occurrences corresponding to the target first record;

[0218] If the difference K2 is greater than 0, K2 second records are added to the second difference record set.

[0219] In a possible implementation, the determining module 1004 is specifically configured to:

[0220] If the first difference record set is not empty, and / or the second difference record set is not empty, it is determined that the to-be-checked data table is inconsistent with the reference data table; or;

[0221] If the first difference record set is empty, and the second difference record set is empty, it is determined that the to-be-checked data table is consistent with the reference data table.

[0222] In a possible implementation, the data table verification device 1000 provided in this embodiment further includes:

[0223] A repair module is used to repair the data table to be checked according to the first difference record set and the second difference record set if it is determined that the data table to be checked is inconsistent with the reference data table, so as to obtain a repaired data table, and the repaired data table is consistent with the reference data table.

[0224] In a possible implementation, the repair module is specifically used to:

[0225] If the first difference record set is not empty, deleting the records in the first difference record set from the to-be-checked data table; and / or,

[0226] If the second difference record set is not empty, then the records in the second difference record set are added to the to-be-checked data table.

[0227] In a possible implementation, the to-be-verified data table is a data table in a first database, and the reference data table is a data table in a second database;

[0228] The first database is a target database corresponding to a preset data migration task, and the second database is a source database corresponding to the preset data migration task; or

[0229] The first database is a standby database, and the second database is a primary database.

[0230] The data table verification device provided in this embodiment can execute the data table verification method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0231] Fig.11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 provided in this embodiment includes: a memory 1101 and at least one processor 1102; a computer program is stored in the memory 1101, and the computer program is configured to be executed by the processor 1102 to implement the data table verification method provided by any of the above method embodiments, and its implementation principle and technical effect are similar, which will not be repeated here.

[0232] Optionally, the memory 1101 may be independent or integrated with the processor 1102. When the memory 1101 is a device independent of the processor 1102, the electronic device 1100 further includes a bus 1103 for connecting the memory 1101 and the processor 1102.

[0233] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement a data table verification method as provided in any of the aforementioned method embodiments. The implementation principle and technical effect are similar and will not be elaborated here.

[0234] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements a data table verification method as provided in any of the aforementioned method embodiments. The implementation principle and technical effect are similar and will not be elaborated here.

[0235] An embodiment of the present application also provides a chip, including: a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to implement the data table verification method provided by any of the above method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

[0236] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), ASICs, etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0237] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0238] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0239] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0240] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device.

[0241] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data table verification method, characterized in that: include: Obtaining a data table to be verified and a reference data table, wherein the data table to be verified and the reference data table have the same table structure; Processing the data table to be verified to obtain a first virtual data table including at least one first record, and determining the first occurrence number of each first record in the first virtual data table in the data table to be verified; wherein each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified; Processing the reference data table to obtain a second virtual data table including at least one second record, and determining the second occurrence number of each second record in the second virtual data table in the reference data table; wherein each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table; Determining whether the to-be-checked data table is consistent with the reference data table according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table; Determining whether the to-be-checked data table and the reference data table are consistent according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table, includes: Generate a first difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the first difference record set includes: M records that are redundant in the to-be-checked data table compared to the reference data table; M is an integer greater than or equal to 0; Generate a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the second difference record set includes: N records missing from the to-be-checked data table compared to the reference data table; N is an integer greater than or equal to 0; According to the first difference record set and the second difference record set, it is determined whether the to-be-checked data table is consistent with the reference data table.

2. The method according to claim 1, characterized in that Generating a first difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, includes: Initialize the first difference record set to be empty; For each first record in the first virtual data table, perform the following steps respectively: Determining in the second virtual data table whether there is a target second record matching the first record; If so, determining a difference K1 between a first number of occurrences corresponding to the first record and a second number of occurrences corresponding to the target second record; If K1 is greater than 0, K1 first records are added to the first difference record set.

3. The method according to claim 2, characterized in that Generating a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, includes: Initialize the second difference record set to be empty; For each second record in the second virtual data table, perform the following steps respectively: Determining in the first virtual data table whether there is a target first record matching the second record; If so, determining a difference K2 between a second number of occurrences corresponding to the second record and a first number of occurrences corresponding to the target first record; If the difference K2 is greater than 0, K2 second records are added to the second difference record set.

4. The method according to any one of claims 1 to 3, characterized in that: Determining whether the to-be-checked data table is consistent with the reference data table according to the first difference record set and the second difference record set includes: If the first difference record set is not empty, and / or the second difference record set is not empty, it is determined that the to-be-checked data table is inconsistent with the reference data table; or; If the first difference record set is empty, and the second difference record set is empty, it is determined that the to-be-checked data table is consistent with the reference data table.

5. The method according to any one of claims 1 to 3, characterized in that: Also includes: If it is determined that the data table to be checked is inconsistent with the reference data table, the data table to be checked is repaired according to the first difference record set and the second difference record set to obtain a repaired data table, and the repaired data table is consistent with the reference data table.

6. The method according to claim 5, characterized in that Repairing the to-be-verified data table according to the first difference record set and the second difference record set to obtain a repaired data table, including: If the first difference record set is not empty, deleting the records in the first difference record set from the to-be-checked data table; and / or, If the second difference record set is not empty, then the records in the second difference record set are added to the to-be-checked data table.

7. The method according to any one of claims 1 to 3 and 6, characterized in that: The to-be-verified data table is a data table in the first database, and the reference data table is a data table in the second database; The first database is a target database corresponding to a preset data migration task, and the second database is a source database corresponding to the preset data migration task; or The first database is a standby database, and the second database is a primary database.

8. A data table verification device, characterized in that: include: An acquisition module, used for acquiring a to-be-checked data table and a reference data table, wherein the to-be-checked data table and the reference data table have the same table structure; A first processing module is used to process the data table to be verified to obtain a first virtual data table including at least one first record, and determine the first occurrence number of each first record in the first virtual data table in the data table to be verified; wherein each first record in the first virtual data table is different from each other, and each first record appears at least once in the data table to be verified; A second processing module is used to process the reference data table to obtain a second virtual data table including at least one second record, and determine the second occurrence number of each second record in the second virtual data table in the reference data table; wherein each second record in the second virtual data table is different from each other, and each second record appears at least once in the reference data table; a determination module, configured to determine whether the data table to be checked is consistent with the reference data table according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table; When determining whether the to-be-checked data table and the reference data table are consistent according to the first virtual data table, the first number of occurrences corresponding to each first record in the first virtual data table, the second virtual data table, and the second number of occurrences corresponding to each second record in the second virtual data table, the determining module is specifically used to: Generate a first difference record set and according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the first difference record set includes: M records that are redundant in the to-be-checked data table compared to the reference data table; M is an integer greater than or equal to 0; Generate a second difference record set according to the first virtual data table, the first occurrence number corresponding to each first record in the first virtual data table, the second virtual data table, and the second occurrence number corresponding to each second record in the second virtual data table, wherein the second difference record set includes: N records missing from the to-be-checked data table compared to the reference data table; N is an integer greater than or equal to 0; According to the first difference record set and the second difference record set, it is determined whether the to-be-checked data table is consistent with the reference data table.

9. An electronic device, characterized in that: include: memory and at least one processor; A computer program is stored in the memory, and the computer program is configured to be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 7 when the computer program is executed by a processor.

Images