Data comparison method and device, electronic equipment and computer storage medium

Abstract

The invention provides a data comparison method and device, electronic equipment and a computer storage medium. In a database set, a database is randomly selected as a reference database; then, tables, exceeding a data volume threshold value, in the reference database serve as reference tables; for each reference table, determining a target table in a target database according to the basic information of the reference table; for each reference table, determining a data queue of the reference table and a data queue of the target table; then, according to the data queue of the reference table and the data queue of the target table, generating a data comparison result; and if the data comparison result shows that the data difference in the reference table and the target table is greater than a difference threshold value, generating early warning information. Therefore, the differences between the tables of the database are effectively compared, the early warning information is generated when the data difference is greater than the difference threshold value, and the test data quality is indirectly improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data comparison method, device, electronic device and computer storage medium. Background Art

[0002] A system generally has multiple deployment environments for testing and online needs, such as functional test environment, non-functional test environment and production environment (production environment refers to the environment where the system is officially launched). If the data distribution of the data tables in the databases of a system in different deployment environments (such as non-functional test environment and production environment) is quite different, it may cause different indexes and returned data amounts when querying or updating the same table (executing the same SQL) in two different environments, which may cause SQL execution problems (such as slow queries and memory overflows) not to be tested or difficult to reproduce after they occur (for example, if they occur in the production environment, when they need to be reproduced in the test environment, the corresponding data needs to be entered in the test environment to make the data distribution characteristics close) to reproduce, thereby taking up a lot of human resources and time to solve the problem, delaying the recovery and use of online functions. Summary of the invention

[0003] In view of this, the present application provides a data comparison method, device, electronic device and computer storage medium, which effectively compare the differences between tables in the database, and generate warning information when the data difference between the tables in the database is greater than the difference threshold, thereby indirectly improving the quality of test data.

[0004] The first aspect of the present application provides a data comparison method, comprising:

[0005] In the database set, a database is randomly selected as a reference database; wherein the database set includes at least two databases;

[0006] Using a table in the benchmark database that exceeds a data volume threshold as a benchmark table;

[0007] For each of the reference tables, a target table is determined in a target database according to basic information of the reference table; wherein the target database is any database in the database set except the reference database;

[0008] For each of the reference tables, determining a data queue of the reference table and determining a data queue of the target table;

[0009] Generate a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than a difference threshold;

[0010] If the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold, a warning message is generated.

[0011] Optionally, the basic information of the reference table includes: the name and structure of the reference table; and for each of the reference tables, determining the target table in the target database according to the basic information of the reference table includes:

[0012] Search the target database according to the name of the reference table to obtain a query result;

[0013] If the query result indicates that there is a table in the target database with the same name as the reference table, determining whether the structure of the pending table is consistent with the structure of the reference table; wherein the pending table is a table in the target database with the same name as the reference table;

[0014] If it is determined that the structure of the pending table is consistent with the structure of the reference table, the pending table is used as the target table.

[0015] Optionally, for each of the reference tables, determining the data queue of the reference table and determining the data queue of the target table includes:

[0016] For each of the reference tables, respectively calculating the standard cardinality of each index of the reference table, and respectively calculating the standard cardinality of each index of the target table;

[0017] generating a data queue of the reference table according to the standard cardinality of each index of the reference table;

[0018] A data queue of the target table is generated according to the standard cardinality of each index of the target table.

[0019] Optionally, respectively calculating the standard cardinality of each index of the reference table includes:

[0020] For each index of the reference table, determining an index cardinality of the index;

[0021] The standard cardinality of the index is determined according to the index cardinality and the total data volume of the index.

[0022] Optionally, respectively calculating the standard cardinality of each index of the target table includes:

[0023] For each index of the target table, determine the index cardinality of the index;

[0024] The standard cardinality of the index is determined according to the index cardinality and the total data volume of the index.

[0025] Optionally, generating a data comparison result according to the data queue of the reference table and the data queue of the target table includes:

[0026] Calculate the distance between the data queue of the reference table and the data queue of the target table according to a preset algorithm to obtain a target distance;

[0027] If the target distance is greater than the distance threshold, a data comparison result is generated in which the data difference between the reference table and the target table is greater than the difference threshold.

[0028] Optionally, the preset algorithm includes a Chebyshev distance calculation method and / or a Manhattan distance algorithm.

[0029] A second aspect of the present application provides a data comparison device, comprising:

[0030] A selection unit, used to arbitrarily select a database in a database set as a reference database; wherein the database set includes at least two databases;

[0031] A first determining unit, configured to use a table in the reference database that exceeds a data volume threshold as a reference table;

[0032] A second determining unit is used to determine a target table in a target database according to basic information of each of the reference tables; wherein the target database is any database in the database set except the reference database;

[0033] A third determining unit, configured to determine, for each of the reference tables, a data queue of the reference table and a data queue of the target table;

[0034] A generating unit, configured to generate a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than a difference threshold;

[0035] The early warning unit is used to generate early warning information if the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold.

[0036] Optionally, the basic information of the reference table includes: the name and structure of the reference table; the second determining unit includes:

[0037] A query unit, used for querying in the target database according to the name of the reference table to obtain a query result;

[0038] a judgment unit, configured to judge whether the structure of the pending table is consistent with the structure of the reference table if the query result indicates that there is a table with the same name as the reference table in the target database; wherein the pending table is a table with the same name as the reference table in the target database;

[0039] The second determining subunit is configured to use the pending table as a target table if the determining unit determines that the structure of the pending table is consistent with the structure of the reference table.

[0040] Optionally, the third determining unit includes:

[0041] a first calculation unit, configured to calculate, for each of the reference tables, a standard cardinality of each index of the reference table and a standard cardinality of each index of the target table;

[0042] a data queue generating unit, configured to generate a data queue of a reference table according to a standard cardinality of each index of the reference table;

[0043] The data queue generating unit is further configured to generate a data queue of the target table according to a standard cardinality of each index of the target table.

[0044] Optionally, when the first calculation unit is used to calculate the standard cardinality of each index of the reference table respectively, it includes:

[0045] A first index cardinality determination unit, configured to determine, for each index of the reference table, an index cardinality of the index;

[0046] The first standard cardinality determination unit is used to determine the standard cardinality of the index according to the index cardinality and the total data volume of the index.

[0047] Optionally, when the first calculation unit is used to calculate the standard cardinality of each index of the target table respectively, it includes:

[0048] A second index cardinality determination unit, configured to determine, for each index of the target table, an index cardinality of the index;

[0049] The second standard cardinality determination unit is used to determine the standard cardinality of the index according to the index cardinality and the total data volume of the index.

[0050] Optionally, the generating unit includes:

[0051] A second calculation unit, configured to calculate the distance between the data queue of the reference table and the data queue of the target table according to a preset algorithm to obtain a target distance;

[0052] The generating subunit is used for generating a data comparison result in which the data difference between the reference table and the target table is greater than the difference threshold value if the target distance is greater than the distance threshold value.

[0053] Optionally, the preset algorithm includes a Chebyshev distance calculation method and / or a Manhattan distance algorithm.

[0054] A third aspect of the present application provides an electronic device, including:

[0055] one or more processors;

[0056] a storage device having one or more programs stored thereon;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement the data comparison method as described in any one of the first aspects.

[0058] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data comparison method as described in any one of the first aspects is implemented.

[0059] As can be seen from the above scheme, the present application provides a data comparison method, device, electronic device and computer storage medium, in a database set, arbitrarily select a database as a reference database; then, use the table in the reference database that exceeds the data volume threshold as the reference table; for each of the reference tables, determine the target table in the target database according to the basic information of the reference table; wherein the target database is any database in the database set except the reference database; for each of the reference tables, determine the data queue of the reference table and determine the data queue of the target table; then, generate a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than the difference threshold; finally, if the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold, then generate a warning message. Thus, the differences between the tables of the database are effectively compared, and when the data difference between the tables of the database is greater than the difference threshold, a warning message is generated, thereby indirectly improving the quality of the test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0061] Figure 1 A specific flow chart of a data comparison method provided in an embodiment of the present application;

[0062] Figure 2 A data queue for determining a reference table and a data queue for determining a target table provided in another embodiment of the present application;

[0063] Figure 3 A schematic diagram of a data comparison device provided in another embodiment of the present application;

[0064] Figure 4 A schematic diagram of an electronic device for implementing a data comparison method provided in another embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0067] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0068] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0069] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0070] A system generally has multiple deployment environments for testing and online needs, such as functional test environment, non-functional test environment and production environment (production environment refers to the environment where the system is officially launched). If the data distribution of the data tables in the databases of a system in different deployment environments (such as non-functional test environment and production environment) is quite different, it may cause different indexes and returned data amounts when querying or updating the same table (executing the same SQL) in two different environments, which may cause SQL execution problems (such as slow queries and memory overflows) not to be tested or difficult to reproduce after they occur (for example, if they occur in the production environment, when they need to be reproduced in the test environment, the corresponding data needs to be entered in the test environment to make the data distribution characteristics close) to reproduce, thereby taking up a lot of human resources and time to solve the problem, delaying the recovery and use of online functions.

[0071] Generally, it is hoped that the data distribution of a system in different deployment environments is as similar as possible, so as to improve the accuracy of the test. The data distribution of the index field of the data table can be regarded as a representative of the data distribution of the table. Therefore, the embodiment of the present application provides a data comparison method, such as Figure 1 As shown, the specific steps include:

[0072] S101. Randomly select a database from a database set as a benchmark database.

[0073] Among them, the database set includes at least two databases. The databases in the database set are databases of different deployment environments of the same system. For example, the databases of the non-functional test environment and the production environment under the same system. If the index data distribution of the same table is different, it may cause different index selectivity. Under the same query conditions, the index used to execute the same SQL may be different, and the amount of data returned by the query may also be quite different, so that the program execution time and memory usage and other parameters may also be quite different, that is, the test results will be distorted, thereby increasing the risk of problems such as slow SQL query or memory overflow.

[0074] S102: Use a table in the benchmark database that exceeds a data volume threshold as a benchmark table.

[0075] S103 . For each reference table, determine a target table in a target database according to basic information of the reference table.

[0076] The target database is any database in the database set except the benchmark database.

[0077] It should be noted that the basic information of the benchmark table includes but is not limited to the name and structure of the benchmark table, which is not limited here.

[0078] Optionally, in another embodiment of the present application, an implementation of step S103 specifically includes the following steps:

[0079] First, query the target database according to the name of the benchmark table to obtain the query result; if the query result shows that there is a table with the same name as the benchmark table in the target database, then determine whether the structure of the pending table is consistent with the structure of the benchmark table; wherein the pending table is a table with the same name as the benchmark table in the target database; if it is determined that the structure of the pending table is consistent with the structure of the benchmark table, then use the pending table as the target table.

[0080] S104 . For each reference table, determine a data queue of the reference table and a data queue of the target table.

[0081] Optionally, in another embodiment of the present application, an implementation of step S104 is as follows: Figure 2 As shown, including:

[0082] S201 . For each reference table, respectively calculate the standard cardinality of each index of the reference table, and respectively calculate the standard cardinality of each index of the target table.

[0083] In the specific implementation process of the present application, the implementation method of respectively calculating the standard cardinality of each index of the reference table includes the following steps:

[0084] First, for each index of the reference table, the index cardinality of the index is determined; then, based on the index cardinality of the index and the total data volume, the standard cardinality of the index is determined.

[0085] It should be noted that the index cardinality of a database index refers to the number of unique values ​​(non-duplicate values) of the field corresponding to the index.

[0086] In MySQL, cardinality can be viewed through SHOW INDEX FROM table name. MySQL uses two APIs to understand the distribution information of the storage engine's index values ​​in order to decide how to use the index: records_in_range() and info(). records_in_range() obtains the approximate number of records in this range by passing two boundary values ​​to the storage engine, while info() returns various types of index statistics, including index cardinality.

[0087] Index cardinality has a significant impact on database performance. When the index cardinality is high, it means that the index column contains many different values, so the index works best and can quickly filter data rows. For example, if a data column contains many different age values, the index will quickly locate the specific data row; but if a data column has only a few different values ​​(such as the gender column has only two values, "M" and "F"), the index will not be effective. Therefore, choosing a column with a high cardinality as an index column can significantly improve query efficiency.

[0088] Specifically, the standard cardinality of the index is determined based on the index cardinality and the total data volume of the index. It can be but is not limited to taking the quotient of the index cardinality and the total data volume as the standard cardinality of the index, which indicates the degree of dispersion of the index cardinality in the entire benchmark table. It can also be understood as the selectivity of this index. The combination of the standard cardinalities of all indexes of a data table (here is the benchmark table) can be used as a quantitative indicator of data distribution.

[0089] It should be noted that the standard cardinality of the index is obviously greater than 0 and less than 1. In this application, the index cardinality is not used directly but the "standard cardinality" value divided by the total number of records in the table is used. This is because the data volume levels of the two target tables may be different, and the standard cardinality takes into account the table data volume. In general, the standard cardinality will also be used as an important basis for index selection in practical applications.

[0090] In the specific implementation process of the present application, the implementation method of respectively calculating the standard cardinality of each index of the target table includes the following steps:

[0091] First, for each index of the target table, determine the index cardinality of the index; then determine the standard cardinality of the index based on the index cardinality and the total data volume.

[0092] It should be noted that the method of respectively calculating the standard cardinality of each index of the target table may refer to the method of respectively calculating the standard cardinality of each index of the reference table, which will not be described in detail here.

[0093] S202 : Generate a data queue of the reference table according to the standard cardinality of each index of the reference table.

[0094] Since a table can have multiple indexes, each index, after the above calculation, obtains a "standard cardinality" numerical queue. The standard cardinality of each index corresponds to an element in the queue. The two tables correspond to two data queues. Obviously, each element is a positive number between 0 and 1.

[0095] In the specific implementation process of the present application, the standard cardinality of each index of the reference table may be, but is not limited to, sorted from large to small to obtain a data queue of the reference table, which is not limited here.

[0096] S203: Generate a data queue of the target table according to the standard cardinality of each index of the target table.

[0097] It is understandable that the method of generating the data queue of the target table according to the standard cardinality of each index of the target table should be consistent with the method of generating the data queue of the reference table according to the standard cardinality of each index of the reference table.

[0098] S105 . Generate a data comparison result according to the data queue of the reference table and the data queue of the target table.

[0099] The data comparison result is whether the data difference between the reference table and the target table is greater than the difference threshold.

[0100] Optionally, in another embodiment of the present application, an implementation of step S105 specifically includes the following steps:

[0101] First, the distance between the data queue of the reference table and the data queue of the target table is calculated according to a preset algorithm to obtain a target distance; if the target distance is greater than a distance threshold, a data comparison result is generated in which the data difference between the reference table and the target table is greater than the difference threshold.

[0102] It can be understood that the data queue of the benchmark table and the data queue of the target table can be regarded as two points on a multidimensional coordinate system. Therefore, the distance between the two points can be calculated using a distance algorithm, thereby converting the data distribution similarity into a quantifiable distance feature between coordinate points for evaluation.

[0103] It is understandable that each preset algorithm corresponds to a distance threshold, and the distance threshold is pre-set and changed by technical personnel and relevant authorized staff, and is not limited here.

[0104] It should be noted that the preset algorithms include but are not limited to Chebyshev distance calculation method, Manhattan distance algorithm, Minkowski distance, etc., which are not limited here.

[0105] In the embodiments of the present application, the Chebyshev distance calculation method and the Manhattan distance algorithm (of course, in the actual application of the present application, one of them can be selected to generate data comparison results) are used as examples for explanation:

[0106] Distance calculation is performed according to the calculation method of Chebyshev distance and Manhattan distance, that is, first calculate the absolute value of the difference between the corresponding elements of the two queues (the so-called corresponding elements are the same index), so as to obtain a new queue (difference queue), calculate the maximum value of the queue, and the value obtained is the Chebyshev distance; calculate the sum of all elements of the queue, and the value obtained is the Manhattan distance, and then divide the Manhattan distance by the number of queue elements to obtain the average Manhattan distance (that is, the Manhattan distance is averaged according to the number of elements), that is:

[0107] Among them, x i Represents the data queue of the base table for the element (i.e. the standard cardinality of the corresponding index), y i The data queue of the target table corresponds to the element (i.e., the standard cardinality of the corresponding index), where n is the number of indexes.

[0108] The obtained Chebyshev distance is compared with the distance threshold corresponding to the Chebyshev distance calculation method to obtain a comparison result. If the Chebyshev distance is greater than the distance threshold corresponding to the Chebyshev distance calculation method, a data comparison result is generated in which the data difference between the reference table and the target table is greater than the difference threshold.

[0109] The obtained average Manhattan distance is compared with the distance threshold corresponding to the Manhattan distance calculation method to obtain a comparison result. If the average Manhattan distance is greater than the distance threshold corresponding to the Manhattan distance calculation method, a data comparison result is generated in which the data difference between the benchmark table and the target table is greater than the difference threshold.

[0110] S106. If the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold, a warning message is generated.

[0111] This helps detect anomalies as early as possible, urges testers to prepare and test test data more accurately, and effectively monitors and improves the data quality of the test environment.

[0112] As can be seen from the above scheme, the present application provides a data comparison method, in a database set, arbitrarily select a database as a benchmark database; then, use the table in the benchmark database that exceeds the data volume threshold as the benchmark table; for each benchmark table, determine the target table in the target database according to the basic information of the benchmark table; wherein the target database is any database in the database set except the benchmark database; for each benchmark table, determine the data queue of the benchmark table and determine the data queue of the target table; then, generate a data comparison result according to the data queue of the benchmark table and the data queue of the target table; wherein the data comparison result is whether the data difference between the benchmark table and the target table is greater than the difference threshold; finally, if the data comparison result shows that the data difference between the benchmark table and the target table is greater than the difference threshold, then generate a warning message. Thus, the differences between the tables of the database are effectively compared, and when the data difference between the tables of the database is greater than the difference threshold, a warning message is generated, thereby indirectly improving the quality of the test data.

[0113] The present application embodiment provides a data comparison device, such as Figure 3 As shown, specifically including:

[0114] The selection unit 301 is used to arbitrarily select a database in the database set as a reference database.

[0115] The database set includes at least two databases.

[0116] The first determining unit 302 is configured to use a table in the reference database that exceeds a data volume threshold as a reference table.

[0117] The second determining unit 303 is configured to determine a target table in the target database according to basic information of each reference table.

[0118] The target database is any database in the database set except the benchmark database.

[0119] Optionally, in another embodiment of the present application, the basic information of the reference table includes: the name and structure of the reference table; the second determining unit 303 includes:

[0120] The query unit is used to query in the target database according to the name of the reference table to obtain the query result.

[0121] The judgment unit is used to judge whether the structure of the pending table is consistent with the structure of the reference table if the query result shows that there is a table with the same name as the reference table in the target database.

[0122] The pending table is a table in the target database with the same name as the reference table.

[0123] The second determining subunit is configured to use the pending table as the target table if the determining unit determines that the structure of the pending table is consistent with the structure of the reference table.

[0124] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0125] The third determining unit 304 is configured to determine, for each reference table, a data queue of the reference table and a data queue of the target table.

[0126] Optionally, in another embodiment of the present application, an implementation of the third determining unit 304 includes:

[0127] The first calculation unit is used to calculate the standard cardinality of each index of each reference table and the standard cardinality of each index of the target table for each reference table.

[0128] The data queue generating unit is used to generate a data queue of the reference table according to a standard cardinality of each index of the reference table.

[0129] The data queue generating unit is further used to generate a data queue of the target table according to the standard cardinality of each index of the target table.

[0130] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.

[0131] Optionally, in another embodiment of the present application, when the first calculation unit is used to calculate the standard cardinality of each index of the reference table respectively, it includes:

[0132] The first index cardinality determination unit is configured to determine the index cardinality of each index of the reference table.

[0133] The first standard cardinality determination unit is used to determine the standard cardinality of the index according to the index cardinality and the total data volume of the index.

[0134] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0135] Optionally, in another embodiment of the present application, when the first calculation unit is used to calculate the standard cardinality of each index of the target table respectively, it includes:

[0136] The second index cardinality determination unit is used to determine the index cardinality of each index of the target table.

[0137] The second standard cardinality determination unit is used to determine the standard cardinality of the index according to the index cardinality and the total data volume of the index.

[0138] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0139] The generating unit 305 is used to generate a data comparison result according to the data queue of the reference table and the data queue of the target table.

[0140] The data comparison result is whether the data difference between the reference table and the target table is greater than the difference threshold.

[0141] Optionally, in another embodiment of the present application, an implementation of the generating unit 305 includes:

[0142] The second calculation unit is used to calculate the distance between the data queue of the reference table and the data queue of the target table according to a preset algorithm to obtain a target distance.

[0143] The generating subunit is used to generate a data comparison result if the target distance is greater than the distance threshold, and the data difference between the reference table and the target table is greater than the difference threshold.

[0144] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0145] Optionally, in another embodiment of the present application, the preset algorithm includes a Chebyshev distance calculation method and / or a Manhattan distance algorithm.

[0146] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0147] The early warning unit 306 is configured to generate early warning information if the data comparison result indicates that the data difference between the reference table and the target table is greater than a difference threshold.

[0148] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0149] As can be seen from the above scheme, the present application provides a data comparison device, wherein the selection unit 301 selects any database in the database set as a reference database; then, the first determination unit 302 selects the table in the reference database that exceeds the data volume threshold as the reference table; the second determination unit 303 determines the target table in the target database according to the basic information of the reference table for each reference table; wherein the target database is any database in the database set except the reference database; the third determination unit 304 determines the data queue of the reference table and the data queue of the target table for each reference table; then, the generation unit 305 generates a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than the difference threshold; finally, if the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold, the warning unit 306 generates a warning message. Thus, the differences between the tables of the database are effectively compared, and when the data difference between the tables of the database is greater than the difference threshold, a warning message is generated, thereby indirectly improving the quality of the test data.

[0150] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0151] Another embodiment of the present application provides an electronic device, such as Figure 4 As shown, including:

[0152] One or more processors 401 .

[0153] The storage device 402 stores one or more programs.

[0154] When the one or more programs are executed by the one or more processors 401 , the one or more processors 401 implement the data comparison method as described in any one of the above embodiments.

[0155] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the data comparison method as described in any one of the above embodiments is implemented.

[0156] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0159] Another embodiment of the present application provides a computer program product. When the computer program product is executed, it is used to perform any of the above data comparison methods.

[0160] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0161] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0162] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0163] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A data comparison method, characterized in that: include: In the database set, a database is randomly selected as a reference database; wherein the database set includes at least two databases; Using a table in the benchmark database that exceeds a data volume threshold as a benchmark table; For each of the reference tables, a target table is determined in a target database according to basic information of the reference table; wherein the target database is any database in the database set except the reference database; For each of the reference tables, determining a data queue of the reference table and determining a data queue of the target table; Generate a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than a difference threshold; If the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold, a warning message is generated.

2. The data comparison method according to claim 1, characterized in that: The basic information of the reference table includes: the name and structure of the reference table; for each of the reference tables, determining the target table in the target database according to the basic information of the reference table includes: Search the target database according to the name of the reference table to obtain a query result; If the query result indicates that there is a table in the target database with the same name as the reference table, determining whether the structure of the pending table is consistent with the structure of the reference table; wherein the pending table is a table in the target database with the same name as the reference table; If it is determined that the structure of the pending table is consistent with the structure of the reference table, the pending table is used as the target table.

3. The data comparison method according to claim 1, characterized in that: The step of determining the data queue of the reference table and the data queue of the target table for each reference table includes: For each of the reference tables, respectively calculating the standard cardinality of each index of the reference table, and respectively calculating the standard cardinality of each index of the target table; generating a data queue of the reference table according to the standard cardinality of each index of the reference table; A data queue of the target table is generated according to the standard cardinality of each index of the target table.

4. The data comparison method according to claim 3, characterized in that: The step of respectively calculating the standard cardinality of each index of the reference table includes: For each index of the reference table, determining an index cardinality of the index; The standard cardinality of the index is determined according to the index cardinality and the total data volume of the index.

5. The data comparison method according to claim 3, characterized in that: The step of respectively calculating the standard cardinality of each index of the target table includes: For each index of the target table, determine the index cardinality of the index; The standard cardinality of the index is determined according to the index cardinality and the total data volume of the index.

6. The data comparison method according to claim 1, characterized in that: The step of generating a data comparison result according to the data queue of the reference table and the data queue of the target table includes: Calculate the distance between the data queue of the reference table and the data queue of the target table according to a preset algorithm to obtain a target distance; If the target distance is greater than the distance threshold, a data comparison result is generated in which the data difference between the reference table and the target table is greater than the difference threshold.

7. The data comparison method according to claim 6, characterized in that: The preset algorithm includes a Chebyshev distance calculation method and / or a Manhattan distance algorithm.

8. A data comparison device, characterized in that: include: A selection unit, used to arbitrarily select a database in a database set as a reference database; wherein the database set includes at least two databases; A first determining unit, configured to use a table in the reference database that exceeds a data volume threshold as a reference table; A second determining unit is used to determine a target table in a target database according to basic information of each of the reference tables; wherein the target database is any database in the database set except the reference database; A third determining unit, configured to determine, for each of the reference tables, a data queue of the reference table and a data queue of the target table; A generating unit, configured to generate a data comparison result according to the data queue of the reference table and the data queue of the target table; wherein the data comparison result is whether the data difference between the reference table and the target table is greater than a difference threshold; The early warning unit is used to generate early warning information if the data comparison result shows that the data difference between the reference table and the target table is greater than the difference threshold.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the data comparison method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the data comparison method as claimed in any one of claims 1 to 7 is implemented.

Images