Data consistency verification method and device, electronic equipment, medium and product

Abstract

The invention provides a data consistency verification method and device, electronic equipment, a medium and a product. The method comprises the following steps: acquiring a built-in function of a source database data object, and obtaining a parameter list of the built-in function according to the built-in function; obtaining a parameter type value range of the parameter list according to a parameter value type of the parameter list; obtaining parameter values in the parameter type value range by adopting a random sampling algorithm according to the parameter type value range; obtaining a first structured query verification statement according to the parameter value; and sending the first structured query verification statement to the target database end to execute verification to obtain a verification result. According to the scheme, the accuracy of a data consistency verification result can be improved.

Description

Technical Field

[0001] The present application relates to the field of database management, and in particular to a data consistency verification method, device, electronic equipment, medium and product. Background Art

[0002] With the development of technology, it is often necessary to upgrade the database from an old version to a new version, or migrate from one database system to another with better performance, lower cost, and higher security. When migrating or converting a database, it is necessary to verify the data consistency.

[0003] At present, some data consistency verification methods usually only verify whether the content and amount of data are consistent; however, the verification results obtained by this verification method are inaccurate and cannot reflect the integrity and reliability of database migration or conversion. Summary of the invention

[0004] The present application provides a data consistency verification method, device, electronic device, medium and product to improve the accuracy of data consistency verification results.

[0005] In a first aspect, an embodiment of the present application provides a data consistency verification method, including: obtaining a built-in function of a source database data object, and obtaining a parameter list of the built-in function according to the built-in function; obtaining a parameter type value range of the parameter list according to the parameter value type of the parameter list; according to the parameter type value range, using a random sampling algorithm to obtain a parameter value within the parameter type value range; obtaining a first structured query verification statement according to the parameter value; sending the first structured query verification statement to a target database end for verification to obtain a verification result.

[0006] In a possible implementation, a parameter list of a built-in function is obtained according to the built-in function; the method includes: obtaining data metadata information of a source database, and generating a built-in function configuration file list of the source database based on the data metadata information of the source database; obtaining a plaintext data definition statement according to a metadata database processing rule of the source database; the metadata database processing rule is used to characterize the rules that data objects should follow during data migration; performing a keyword comparison between the plaintext data definition statement and the built-in function configuration file list, and using the parameter list in the plaintext data definition statement of the built-in function having the same comparison result as the parameter list of the built-in function.

[0007] In a possible implementation, a random sampling algorithm is used to obtain a parameter value within the parameter type value range according to the parameter type value range; including: determining a target distribution of the random sampling algorithm according to the parameter type value range; selecting a proposed distribution according to the target distribution; selecting a boundary value of the parameter type as the current state of the proposed distribution; based on the current state, iterating until a preset number of iterations is reached to generate a sample sequence; and using the sample values ​​of the sample sequence as parameter values ​​within the parameter type value range.

[0008] In a possible implementation, based on the current state, iteration is performed until a preset number of iterations is reached to generate a sample sequence; including: based on the proposed distribution, calculating the reception probability; the reception probability represents the ratio of the probability of the current state to the candidate state and the inverse process thereof; randomly generating a value in the interval [0, 1], if the reception probability is not less than the randomly generated value, the candidate state is used as the current state of the next iteration; otherwise, the current state is maintained for the next iteration; the current state of each iteration step is obtained, and a sample sequence is generated based on the current state.

[0009] In a possible implementation, obtaining a first structured query verification statement according to parameter values ​​includes: concatenating the parameter values ​​to obtain the first structured query verification statement.

[0010] In a possible implementation, the method further includes: obtaining a second structured query verification statement according to a boundary value of a parameter type value range; and sending the second structured query verification statement to a target database end for verification to obtain a verification result.

[0011] In a possible implementation, obtaining a second structured query verification statement according to a boundary value of a parameter type value range includes: concatenating the boundary values ​​of the parameter type value range to obtain the second structured verification statement.

[0012] In a possible implementation, the method further includes: if the parameter list is a constant, concatenating the constants to obtain a third structured query verification statement.

[0013] In a second aspect, the present application provides a data consistency verification device, which includes: an acquisition module, used to acquire a built-in function of a source database data object, and obtain a parameter list of the built-in function according to the built-in function; a sampling module, used to obtain a parameter type value range of a parameter list according to the parameter value type of the parameter list; according to the parameter type value range, a random sampling algorithm is used to obtain a parameter value within the parameter type value range; a verification module, used to obtain a first structured query verification statement according to the parameter value; the first structured query verification statement is sent to the target database end for verification to obtain a verification result.

[0014] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the above method.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program, which is used to implement the above method when executed by a processor.

[0017] In the data consistency verification method, device, electronic device, medium and product provided by the present application, the built-in function of the source database data object is obtained, and the parameter list of the built-in function is obtained based on the built-in function of the source database; according to the parameter value type of the parameter list, the parameter type value range of the parameter list is obtained; according to the parameter type value range, a random sampling algorithm is used to obtain the parameter value within the parameter type value range; according to the parameter value, a first structured query verification statement is obtained; the first structured query verification statement is sent to the target database end for verification to obtain a verification result. The scheme of the present application obtains the parameter value within the value range of the built-in function parameter type of the source database based on a random sampling algorithm, and generates a verification statement for performing data consistency verification based on the obtained parameter value; on the basis of the basic data consistency verification, a consistency verification on the built-in function is added to make the verification result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] Figure 1 A schematic diagram of a process flow of a data consistency verification method is exemplarily shown;

[0020] Figure 2 A schematic diagram of a process flow of a data consistency verification method is exemplarily shown;

[0021] Figure 3 The logical architecture diagram for data migration of this application;

[0022] Figure 4 A schematic diagram of a process flow of a data consistency verification method is exemplarily shown;

[0023] Figure 5 A flow chart of the process of generating a verification statement for this application;

[0024] Figure 6 A schematic diagram of the structure of a data consistency verification device is exemplarily shown;

[0025] Figure 7 A structural schematic diagram of an electronic device is exemplarily shown in FIG.

[0026] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0028] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings in this application are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable where appropriate, for example, they can be implemented in an order other than those given in the diagram or description of the embodiment of this application. The terms "including" and "having" in the specification and claims and the above-mentioned drawings in this application and any of their variations are intended to cover but not exclude inclusion, for example, a product or device containing a series of components is not necessarily limited to those components clearly listed, but may include other components that are not clearly listed or inherent to these products or devices. The term "module" used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware or / and software code that can perform functions related to the element.

[0029] With the development of technology, it is often necessary to upgrade the database from an old version to a new version, or migrate from one database system to another with better performance, lower cost, and higher security. When migrating or converting a database, it is necessary to verify the data consistency.

[0030] At present, some data consistency verification methods usually only verify whether the content and amount of data are consistent; in actual application, due to the differences in data types and conversion rules between heterogeneous databases, database objects such as views, stored procedures, and functions defined by built-in functions or expressions in the database system may fail to report errors even though they have been successfully migrated. The verification results obtained using conventional verification methods are inaccurate and cannot reflect the integrity and reliability of database migration or conversion.

[0031] The technical content provided by this application is intended to solve some technical problems of related technologies such as those mentioned above. In the data consistency verification method, device, electronic device, medium and product of this application, the built-in function of the source database data object is obtained, and the parameter list of the built-in function is obtained based on the built-in function of the source database; according to the parameter value type of the parameter list, the parameter type value range of the parameter list is obtained; according to the parameter type value range, a random sampling algorithm is used to obtain the parameter value within the parameter type value range; according to the parameter value, a first structured query verification statement is obtained; the first structured query verification statement is sent to the target database end for verification to obtain a verification result. The scheme of this application obtains the parameter value within the value range of the built-in function parameter type of the source database based on a random sampling algorithm, and generates a verification statement for performing data consistency verification based on the obtained parameter value; on the basis of conventional data consistency verification, a consistency check on the built-in function is added to make the verification result more accurate.

[0032] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0033] Embodiment 1

[0034] Figure 1 A schematic diagram of a data consistency verification method is shown as an example. Figure 1 As shown, the method includes:

[0035] Step 101: Obtain a built-in function of a source database data object, and obtain a parameter list of the built-in function according to the built-in function;

[0036] Step 102: According to the parameter value type of the parameter list, obtain the parameter type value range of the parameter list; according to the parameter type value range, use a random sampling algorithm to obtain the parameter value within the parameter type value range;

[0037] Step 103: Obtain a first structured query verification statement according to the parameter value; send the first structured query verification statement to the target database end for verification, and obtain a verification result.

[0038] Specifically, the database migration involves data consistency verification; in actual applications, the database field uses migration tools to complete the migration of the database; the source database (the database system currently using or storing the data to be migrated) and the target database (the destination database system for the migrated data) involved in the database migration are heterogeneous databases; among them, heterogeneous databases refer to multiple database systems with differences in database management system types, data models, data storage and retrieval mechanisms, query languages, etc.; the database migration steps usually include: first, it is necessary to ensure that the source database and the target database system have been installed and configured; then determine which data objects (such as tables, views, stored procedures, etc.) need to be migrated; then use the architecture conversion function of the migration tool to convert the table structure, index, view, etc. of the source database to the format of the target database; perform full or incremental data synchronization during migration; after the database migration is completed, perform comprehensive data verification on the target database, including data integrity, data consistency, etc. The data consistency verification of the database migration in this example can be verified after the database migration is completed, or it can be verified during the database migration process, and this example does not limit this. In this example, when performing data consistency check on the database, the built-in functions of the data objects of the source database must first be obtained, and the parameter list of the built-in functions must be obtained based on the built-in functions; specifically, there will be built-in functions in the data objects in the database, and the data objects refer to views, tables, stored procedures, custom functions, etc.; built-in functions include string functions, numerical functions, aggregate functions, conditional functions, etc.; built-in functions provide functions such as operations, calculations, and conversions on database data. According to the built-in functions, the parameter list of the built-in function parameter type can be obtained. Furthermore, according to the parameter value type of the parameter list, the parameter type value range of the parameter list can be obtained; according to the value range of the parameter type, the parameter type boundary value can be obtained, and the maximum boundary value and the minimum boundary value of the parameter type are used as the initial value of the random sampling algorithm, and the sample value is obtained by sampling within the value range of the parameter type, and the sample is used as the parameter value. This example does not limit the type of the random sampling algorithm, as long as the sampling range covers the value range of the database parameter type. Specifically, a random sampling algorithm refers to an algorithm that does not impose any human will on the sampling location or time when taking samples, but instead draws samples from the population according to a certain random principle; this algorithm can ensure that samples from any part have the opportunity to be taken, and that the probability of each sample being selected is the same.Further, the first structured query verification statement is obtained according to the obtained parameter value; the obtained structured query verification statement is sent to the target database end for verification, and the data consistency verification result can be obtained; after the data consistency verification result is obtained, the verification result is synchronized to the migration report; in actual application, the migration report contains a description of the data consistency verification method, such as the comparison check of query data between the new and old systems, the comparison result after restoring data and supplementing the last day's business, etc.; as well as specific indicators and results of data consistency verification, such as integrity, consistency, total score balance, number of records and inspection results of special sample data, etc. In this example, a random sampling algorithm is used to sample the parameter type value range, and the obtained sample value is used as the parameter value, and then a structured query verification statement is generated according to the parameter value and sent to the target database end for verification; because the generated structured query verification statement includes the parameter value of the built-in function, the verification scope for the database is more comprehensive, the generated structured query verification statement is more accurate, and the verification result obtained by executing the verification statement is more accurate.

[0039] Furthermore, it is necessary to obtain the parameter list of the built-in function according to the source database; accordingly, as an example, Figure 2 A flow chart of a data consistency verification method is exemplarily shown; based on any example, a parameter list of a built-in function is obtained according to the built-in function, specifically including:

[0040] Step 201: Obtain data meta information of a source database, and generate a list of built-in function configuration files of the source database based on the data meta information of the source database;

[0041] Step 202: Obtain a plaintext data definition statement according to the metadata database processing rule of the source database; the metadata database processing rule is used to represent the rules that the data object should follow during data migration;

[0042] Step 203: perform keyword comparison between the plain text data definition statement and the built-in function configuration file list, and use the parameter list in the plain text data definition statement of the built-in function with the consistent comparison result as the parameter list of the built-in function.

[0043] Specifically, when it is necessary to obtain the parameter list of the built-in function according to the built-in function, first, the metadata of the source database needs to be queried; the configuration file of the source database is generated according to the obtained metadata; the configuration file stores the key information of the built-in functions commonly used by the source database, such as the function name, the parameter type of the function, etc. Next, according to the metadata database processing rules of the source database, the plaintext data definition statement corresponding to the source database is obtained; wherein the metadata database processing rules represent the migration rules that the data object should follow during data migration; including data mapping rules, default value mapping rules, field mapping rules, table mapping rules, etc. Metadata database processing rules are intended to ensure the accuracy, consistency, reliability and security of metadata; they are formulated according to the needs of the specific metadata management system and database. Specifically, a plaintext data definition statement is generated according to the metadata database processing rules; wherein the plaintext data definition statement refers to a data definition statement that is not encrypted, not encoded, or not hidden in any form. Further, the plaintext definition statement is compared with the built-in function configuration file list by keywords, and the parameter list in the plaintext definition statement of the built-in function with the consistent comparison result is used as the parameter list of the built-in function. In practical applications, the comparison is performed according to keywords to improve the efficiency of obtaining built-in functions. Figure 3 This is the logical architecture diagram for database migration for this application; Figure 3 As shown, when the database needs to be migrated, an operating system for database migration is required, and the migration of the database is completed on the operating system. The front end of the operating system includes a terminal display that can display the data information of the database; and an application programming (API) interface for receiving and transmitting data; the migration process needs to read data from the source database, read the data to be migrated and the metadata of the source database; the metadata in the source database is processed by metadata to obtain the corresponding plaintext data definition statement; the data to be migrated in the source database is processed to obtain the corresponding data; the processed data and the plaintext data definition statement are written into the target database. When verifying the data consistency of the data migration, a structured query verification statement is generated based on the parameter list of the plaintext data definition statement; and the structured query verification statement is subsequently sent to the database migration system for data consistency verification. In this example, the keyword in the plaintext data definition statement of the source database is compared with the keyword in the built-in function configuration list to obtain the parameter list of the built-in function, and then the structured query verification statement for data consistency verification is obtained according to the parameter value within the value range of the parameter list, thereby improving the accuracy of the verification result.

[0044] Specifically, a random sampling algorithm is needed to obtain parameter values; accordingly, as an example, Figure 4A flow chart of a data consistency verification method is exemplified; based on any example, according to the parameter type value range, a random sampling algorithm is used to obtain a parameter value within the parameter type value range, specifically including:

[0045] Step 301: Determine the target distribution of the random sampling algorithm according to the parameter type value range;

[0046] Step 302: Select a proposed distribution according to the target distribution; select a boundary value of the parameter type as the current state of the proposed distribution;

[0047] Step 303: Based on the current state, iterate until a preset number of iterations is reached to generate a sample sequence; and use the sample values ​​of the sample sequence as parameter values ​​within the value range of the parameter type.

[0048] Specifically, in this example, there is no restriction on the type of random sampling algorithm. When using a random sampling algorithm for sampling, it is necessary to determine the target distribution of the random sampling algorithm based on the value range of the parameter type mentioned above. This target distribution represents the value range of the parameter type. Next, determine the proposed distribution based on the target distribution. Generally, the proposed distribution should ensure that the random sampling algorithm can efficiently explore the high-probability area of ​​the target distribution. The boundary value of the parameter type obtained above is used as the initial value of the random sampling algorithm. The initial value is the current state of the proposed distribution. Based on the current state, iterate until the preset number of iterations is reached. At this time, generate a sample sequence generated by each iteration step, and use the sample value of the generated sample sequence as the parameter value.

[0049] Furthermore, it is necessary to generate a sample sequence based on certain selection rules; accordingly, as an example, based on any example, based on the current state, iteration is performed until a preset number of iterations is reached to generate a sample sequence; specifically, the method includes:

[0050] Based on the proposed distribution, the acceptance probability is calculated; the acceptance probability represents the ratio of the probability of the current state to the candidate state and its reverse process;

[0051] A value is randomly generated in the interval [0, 1]. If the receiving probability is not less than the randomly generated value, the candidate state is used as the current state for the next iteration; otherwise, the current state is maintained for the next iteration;

[0052] Get the current state of each iteration and generate a sample sequence based on the current state.

[0053] Combined with the above example, in one example, after selecting the proposed distribution according to the target distribution, the candidate points of the random sampling algorithm are generated based on the proposed distribution, and the candidate points are the sample values ​​of this application. The boundary value of the parameter list value range is used as the current state, and iteration is performed to generate a new sample point for each step; when selecting the sample point, the reception probability is calculated, and the reception probability represents the ratio of the probability of going from the current state to the proposed state to the probability of the opposite process; then, in the interval [0, 1], a numerical value is randomly generated, and the reception probability is compared with the randomly selected numerical value. If the reception probability is greater than or equal to the randomly generated numerical value, the candidate state is used as the current state of the next iteration, and the candidate state represents the sample value of this iteration; if the reception probability is less than the randomly generated numerical value, the current state is maintained for the next iteration; the above process is repeated until the preset number of iterations is reached and the iteration is stopped. By obtaining the current state of each step, a sample sequence can be generated. After obtaining the generated sample sequence, the initial few samples are ignored when obtaining the sample values ​​to reduce the deviation; this is because in the early stage of generating the sample sequence, due to system initialization, algorithm startup or other factors, the generated samples may be unstable or unrepresentative, so these samples may introduce deviations and affect the accuracy of subsequent analysis. In this example, a random sampling algorithm is used to obtain reference values; a structured query verification statement is generated based on the reference value, which is subsequently used to verify data consistency and improve the accuracy of the verification results.

[0054] Based on the above example, in one example, a first structured query verification statement is obtained according to the parameter value; specifically including:

[0055] The parameter values ​​are concatenated to obtain the first structured query verification statement.

[0056] In this example, after obtaining the parameter values ​​according to the random sampling algorithm, the obtained parameter values ​​are concatenated to generate the first structured query verification statement. The first structured query verification statement is then sent to the target database end for verification to obtain the verification result. In this example, the structured query verification statement generated based on the parameter value of the built-in function is used to perform data consistency verification, and the obtained verification result is more accurate.

[0057] Further, based on the above example, in one example, the method further includes:

[0058] A second structured query verification statement is obtained according to the boundary value of the parameter type value range;

[0059] The second structured query verification statement is sent to the target database end for verification to obtain a verification result.

[0060] Specifically, in actual application scenarios, when performing data consistency verification on database migration, in addition to generating verification statements based on the parameter values ​​obtained above, verification statements are also generated based on the boundary values ​​of the parameter type's value range; a corresponding second structured query verification statement is obtained based on the boundary values ​​of the parameter type; the second structured verification statement is also used as a verification statement for data consistency verification; the second structured query verification statement is sent to the target database for verification to obtain a verification result. In this example, the parameter type boundary value is also used to generate a structured query verification statement, further improving the accuracy of the data verification result.

[0061] Based on the above example, in another example, a second structured query verification statement is obtained according to the boundary value of the parameter type value range; including:

[0062] The boundary values ​​of the parameter type value range are concatenated to obtain the second structured verification statement.

[0063] Specifically, after determining the parameter type boundary value according to the value range of the built-in function parameter type, the parameter type boundary values ​​are concatenated to obtain a second structured verification statement. In actual applications, all parameter type boundary values ​​in the database are concatenated to obtain a second structured verification statement for verification, and data consistency verification is performed based on the obtained verification statement, further improving the accuracy of the verification result.

[0064] Based on the above examples, in any example, the method further includes:

[0065] If the parameter list is a constant, the constants are concatenated to obtain the third structured query verification statement.

[0066] Specifically, if the parameter type of the built-in function is a constant, the constant is directly spliced ​​to obtain the third structured query verification statement. Similarly, after obtaining the third structured query verification statement, the third structured query verification statement is sent to the target database end to perform data consistency verification to obtain a verification result. In this example, when the parameter list is a constant, the third structured query verification statement is directly spliced ​​according to the constant, so that the verification result is more accurate.

[0067] Combining the above examples, Figure 5 A flow chart of generating a verification statement for this application; Figure 5As shown, first, the metadata of the source database is obtained, and the plaintext data definition statement of the source database is generated according to the metadata; at the same time, a parameter type configuration file is generated according to the obtained metadata; the keyword comparison between the parameter type configuration file and the plaintext data definition statement is performed to determine whether the keywords are consistent, and a parameter list with consistent keywords is obtained based on this; next, the parameter type is determined according to the parameter list, and if the parameter type of the parameter list is a constant, the constant is directly concatenated to obtain a verification statement; if the parameter type of the parameter list is not a constant, the parameter value of the built-in function is obtained according to the random sampling algorithm, and a verification statement is generated according to the parameter value; after the verification statement is generated, the verification is performed using the verification statement.

[0068] In the data consistency verification method provided by the present embodiment, the built-in function of the source database data object is obtained, and the parameter list of the built-in function is obtained based on the built-in function of the source database; according to the parameter value type of the parameter list, the parameter type value range of the parameter list is obtained; according to the parameter type value range, a random sampling algorithm is used to obtain the parameter value within the parameter type value range; according to the parameter value, a first structured query verification statement is obtained; the first structured query verification statement is sent to the target database end for verification to obtain a verification result. The scheme of the present application obtains the parameter value within the value range of the built-in function parameter type of the source database based on a random sampling algorithm, and generates a verification statement for performing data consistency verification based on the obtained parameter value; on the basis of the basic data consistency verification, a consistency verification on the built-in function is added to make the verification result more accurate.

[0069] Embodiment 2

[0070] Figure 6 A schematic diagram of the structure of a data consistency verification device is shown as an example; Figure 6 As shown, the device comprises:

[0071] An acquisition module 21 is used to acquire a built-in function of a data object in a source database, and obtain a parameter list of the built-in function according to the built-in function;

[0072] The sampling module 22 is used to obtain the parameter type value range of the parameter list according to the parameter value type of the parameter list; according to the parameter type value range, a random sampling algorithm is used to obtain the parameter value within the parameter type value range;

[0073] The verification module 23 is used to obtain a first structured query verification statement according to the parameter value; send the first structured query verification statement to the target database end for verification, and obtain a verification result.

[0074] Specifically, the database migration involves data consistency verification; in actual applications, the database field uses migration tools to complete the migration of the database; the source database (the database system currently using or storing the data to be migrated) and the target database (the destination database system for the migrated data) involved in the database migration are heterogeneous databases; among them, heterogeneous databases refer to multiple database systems with differences in database management system types, data models, data storage and retrieval mechanisms, query languages, etc.; the database migration steps usually include: first, it is necessary to ensure that the source database and the target database system have been installed and configured; then determine which data objects (such as tables, views, stored procedures, etc.) need to be migrated; then use the architecture conversion function of the migration tool to convert the table structure, index, view, etc. of the source database to the format of the target database; perform full or incremental data synchronization during migration; after the database migration is completed, perform comprehensive data verification on the target database, including data integrity, data consistency, etc. The data consistency verification of the database migration in this example can be verified after the database migration is completed, or it can be verified during the database migration process, and this example does not limit this. In this example, when performing data consistency check on the database, the built-in functions of the data objects of the source database must first be obtained, and the parameter list of the built-in functions must be obtained based on the built-in functions; specifically, there will be built-in functions in the data objects in the database, and the data objects refer to views, tables, stored procedures, custom functions, etc.; built-in functions include string functions, numerical functions, aggregate functions, conditional functions, etc.; built-in functions provide functions such as operations, calculations, and conversions on database data. According to the built-in functions, the parameter list of the built-in function parameter type can be obtained. Furthermore, according to the parameter value type of the parameter list, the parameter type value range of the parameter list can be obtained; according to the value range of the parameter type, the parameter type boundary value can be obtained, and the maximum boundary value and the minimum boundary value of the parameter type are used as the initial value of the random sampling algorithm, and the sample value is obtained by sampling within the value range of the parameter type, and the sample is used as the parameter value. This example does not limit the type of the random sampling algorithm, as long as the sampling range is guaranteed to cover the value range of the database parameter type. Specifically, a random sampling algorithm refers to an algorithm that does not impose any human will on the sampling location or time when taking samples, but instead draws samples from the population according to a certain random principle; this algorithm can ensure that samples from any part have the opportunity to be taken, and that the probability of each sample being selected is the same.Further, the first structured query verification statement is obtained according to the obtained parameter value; the obtained structured query verification statement is sent to the target database end for verification, and the data consistency verification result can be obtained; after the data consistency verification result is obtained, the verification result is synchronized to the migration report; in actual application, the migration report contains a description of the data consistency verification method, such as the comparison check of query data between the new and old systems, the comparison result after restoring data and supplementing the last day's business, etc.; as well as specific indicators and results of data consistency verification, such as integrity, consistency, total score balance, number of records and inspection results of special sample data, etc. In this example, a random sampling algorithm is used to sample the parameter type value range, and the obtained sample value is used as the parameter value, and then a structured query verification statement is generated according to the parameter value and sent to the target database end for verification; because the generated structured query verification statement includes the parameter value of the built-in function, the verification scope for the database is more comprehensive, the generated structured query verification statement is more accurate, and the verification result obtained by executing the verification statement is more accurate.

[0075] Furthermore, it is necessary to obtain a parameter list of the built-in function according to the source database; accordingly, as an example, the acquisition module 21 is specifically used for:

[0076] Obtain data metadata of the source database, and generate a list of built-in function configuration files of the source database based on the data metadata of the source database;

[0077] Obtaining a plain text data definition statement according to the metadata database processing rule of the source database; the metadata database processing rule is used to represent the rules that the data object should follow during data migration;

[0078] The plain text data definition statement is compared with the built-in function configuration file list for keywords, and the parameter list in the plain text data definition statement of the built-in function with the same comparison result is used as the parameter list of the built-in function.

[0079] Specifically, when it is necessary to obtain the parameter list of the built-in function according to the built-in function, first, the metadata of the source database needs to be queried; the configuration file of the source database is generated according to the obtained metadata; the configuration file stores the key information of the built-in functions commonly used by the source database, such as the function name, the parameter type of the function, etc. Next, according to the metadata database processing rules of the source database, the plaintext data definition statement corresponding to the source database is obtained; wherein the metadata database processing rules represent the migration rules that the data object should follow during data migration; including data mapping rules, default value mapping rules, field mapping rules, table mapping rules, etc. Metadata database processing rules are intended to ensure the accuracy, consistency, reliability and security of metadata; they are formulated according to the needs of the specific metadata management system and database. Specifically, a plaintext data definition statement is generated according to the metadata database processing rules; wherein the plaintext data definition statement refers to a data definition statement that is not encrypted, not encoded, or not hidden in any form. Further, the plaintext definition statement is compared with the built-in function configuration file list by keywords, and the parameter list in the plaintext definition statement of the built-in function with the consistent comparison result is used as the parameter list of the built-in function. In practical applications, the comparison is performed according to keywords to improve the efficiency of obtaining built-in functions. In this example, the keyword in the plain text data definition statement of the source database is compared with the keyword in the built-in function configuration list to obtain the parameter list of the built-in function. Subsequently, the structured query verification statement for data consistency verification is obtained based on the parameter values ​​within the value range of the parameter list, thereby improving the accuracy of the verification result.

[0080] Specifically, a random sampling algorithm needs to be used to obtain parameter values; accordingly, as an example, the sampling module 22 is specifically used for:

[0081] Determine the target distribution of the random sampling algorithm based on the value range of the parameter type;

[0082] Select a proposed distribution according to the target distribution; select the boundary value of the parameter type as the current state of the proposed distribution;

[0083] Based on the current state, iteration is performed until a preset number of iterations is reached to generate a sample sequence; the sample value of the sample sequence is used as the parameter value within the value range of the parameter type.

[0084] Specifically, in this example, there is no restriction on the type of random sampling algorithm. When using a random sampling algorithm for sampling, it is necessary to determine the target distribution of the random sampling algorithm based on the value range of the parameter type mentioned above. This target distribution represents the value range of the parameter type. Next, determine the proposed distribution based on the target distribution. Generally, the proposed distribution should ensure that the random sampling algorithm can efficiently explore the high-probability area of ​​the target distribution. The boundary value of the parameter type obtained above is used as the initial value of the random sampling algorithm. The initial value is the current state of the proposed distribution. Based on the current state, iterate until the preset number of iterations is reached. At this time, generate a sample sequence generated by each iteration step, and use the sample value of the generated sample sequence as the parameter value.

[0085] Furthermore, it is necessary to generate a sample sequence based on certain selection rules; accordingly, as an example, the sampling module 22 is also specifically used for:

[0086] Based on the proposed distribution, the acceptance probability is calculated; the acceptance probability represents the ratio of the probability of the current state to the candidate state and its reverse process;

[0087] A value is randomly generated in the interval [0, 1]. If the receiving probability is not less than the randomly generated value, the candidate state is used as the current state for the next iteration; otherwise, the current state is maintained for the next iteration;

[0088] Get the current state of each iteration and generate a sample sequence based on the current state.

[0089] Combined with the above example, in one example, after selecting the proposed distribution according to the target distribution, the candidate points of the random sampling algorithm are generated based on the proposed distribution, and the candidate points are the sample values ​​of this application. The boundary value of the parameter list value range is used as the current state, and iteration is performed to generate a new sample point for each step; when selecting the sample point, the reception probability is calculated, and the reception probability represents the ratio of the probability of going from the current state to the proposed state to the probability of the opposite process; then, in the interval [0, 1], a numerical value is randomly generated, and the reception probability is compared with the randomly selected numerical value. If the reception probability is greater than or equal to the randomly generated numerical value, the candidate state is used as the current state of the next iteration, and the candidate state represents the sample value of this iteration; if the reception probability is less than the randomly generated numerical value, the current state is maintained for the next iteration; the above process is repeated until the preset number of iterations is reached and the iteration is stopped. By obtaining the current state of each step, a sample sequence can be generated. After obtaining the generated sample sequence, the initial few samples are ignored when obtaining the sample values ​​to reduce the deviation; this is because in the early stage of generating the sample sequence, due to system initialization, algorithm startup or other factors, the generated samples may be unstable or unrepresentative, so these samples may introduce deviations and affect the accuracy of subsequent analysis. In this example, a random sampling algorithm is used to obtain reference values; a structured query verification statement is generated based on the reference value, which is subsequently used to verify data consistency and improve the accuracy of the verification results.

[0090] Based on the above example, in one example, a first structured query verification statement is obtained according to the parameter value; specifically including:

[0091] The parameter values ​​are concatenated to obtain the first structured query verification statement.

[0092] In this example, after obtaining the parameter values ​​according to the random sampling algorithm, the obtained parameter values ​​are concatenated to generate the first structured query verification statement. The first structured query verification statement is then sent to the target database end for verification to obtain the verification result. In this example, the structured query verification statement generated based on the parameter value of the built-in function is used to perform data consistency verification, and the obtained verification result is more accurate.

[0093] Further, based on the above example, in one example, the verification module 23 is further used to:

[0094] A second structured query verification statement is obtained according to the boundary value of the parameter type value range;

[0095] The second structured query verification statement is sent to the target database end for verification to obtain a verification result.

[0096] Specifically, in actual application scenarios, when performing data consistency verification on database migration, in addition to generating verification statements based on the parameter values ​​obtained above, verification statements are also generated based on the boundary values ​​of the parameter type's value range; a corresponding second structured query verification statement is obtained based on the boundary values ​​of the parameter type; the second structured verification statement is also used as a verification statement for data consistency verification; the second structured query verification statement is sent to the target database for verification to obtain a verification result. In this example, the parameter type boundary value is also used to generate a structured query verification statement, further improving the accuracy of the data verification result.

[0097] Based on the above example, in another example, a second structured query verification statement is obtained according to the boundary value of the parameter type value range; including:

[0098] The boundary values ​​of the parameter type value range are concatenated to obtain the second structured verification statement.

[0099] Specifically, after determining the parameter type boundary value according to the value range of the built-in function parameter type, the parameter type boundary values ​​are concatenated to obtain a second structured verification statement. In actual applications, all parameter type boundary values ​​in the database are concatenated to obtain a second structured verification statement for verification, and data consistency verification is performed based on the obtained verification statement, further improving the accuracy of the verification result.

[0100] Based on the above examples, in any example, the verification module 23 is further used to:

[0101] If the parameter list is a constant, the constants are concatenated to obtain the third structured query verification statement.

[0102] Specifically, if the parameter type of the built-in function is a constant, the constant is directly spliced ​​to obtain the third structured query verification statement. Similarly, after obtaining the third structured query verification statement, the third structured query verification statement is sent to the target database end to perform data consistency verification to obtain a verification result. In this example, when the parameter list is a constant, the third structured query verification statement is directly spliced ​​according to the constant, so that the verification result is more accurate.

[0103] The data consistency verification device provided by this embodiment obtains the built-in function of the source database data object, and obtains the parameter list of the built-in function based on the built-in function of the source database; obtains the parameter type value range of the parameter list according to the parameter value type of the parameter list; obtains the parameter value within the parameter type value range using a random sampling algorithm according to the parameter type value range; obtains the first structured query verification statement according to the parameter value; sends the first structured query verification statement to the target database end for verification to obtain a verification result. The scheme of the present application obtains the parameter value within the value range of the built-in function parameter type of the source database based on a random sampling algorithm, and generates a verification statement for performing data consistency verification according to the obtained parameter value; on the basis of the basic data consistency verification, a consistency verification on the built-in function is added to make the verification result more accurate.

[0104] Embodiment 3

[0105] Figure 7 exemplarily shows a structural schematic diagram of an electronic device, the device comprising:

[0106] The device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, the memory 292, and the communication interface 293 may communicate with each other through the bus 294. The communication interface 293 may be used for information transmission. The processor 291 may call the logic instructions in the memory 292 to execute the above-mentioned example method.

[0107] In addition, the logic instructions in the above-mentioned memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0108] The memory 292 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, that is, implementing the methods in the above method examples.

[0109] The memory 292 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 292 may include a high-speed random access memory and may also include a non-volatile memory.

[0110] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method in any embodiment.

[0111] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the method in any embodiment when executed by a processor.

[0112] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0113] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed in the same time period, but can be executed in different time periods, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0114] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0115] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0116] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0117] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only.

[0120] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

[0121] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variation, use or adaptation of the present invention, which follows the general principles of the present invention and includes common knowledge or conventional techniques in the art not disclosed by the present invention, is not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof.

Claims

1. A data consistency verification method, characterized in that: The method comprises: Acquire a built-in function of a data object in a source database, and obtain a parameter list of the built-in function according to the built-in function; According to the parameter value type of the parameter list, obtaining the parameter type value range of the parameter list; according to the parameter type value range, using a random sampling algorithm to obtain the parameter value within the parameter type value range; A first structured query verification statement is obtained according to the parameter value; the first structured query verification statement is sent to the target database end for verification to obtain a verification result.

2. The method according to claim 1, characterized in that The step of obtaining a parameter list of the built-in function according to the built-in function includes: Acquire data meta information of the source database, and generate a list of built-in function configuration files of the source database based on the data meta information of the source database; Obtaining a plaintext data definition statement according to the metadata database processing rule of the source database; the metadata database processing rule is used to represent the rules that the data object should follow during data migration; A keyword comparison is performed between the plaintext data definition statement and the built-in function configuration file list, and the parameter list in the plaintext data definition statement of the built-in function with consistent comparison results is used as the parameter list of the built-in function.

3. The method according to claim 1, characterized in that The step of obtaining a parameter value within the parameter type value range by using a random sampling algorithm according to the parameter type value range includes: Determining a target distribution of the random sampling algorithm according to a value range of the parameter type; Selecting a proposed distribution according to the target distribution; selecting a boundary value of the parameter type as the current state of the proposed distribution; Based on the current state, iteration is performed until a preset number of iterations is reached to generate a sample sequence; and the sample values ​​of the sample sequence are used as parameter values ​​within the value range of the parameter type.

4. The method according to claim 3, characterized in that The step of iterating based on the current state until a preset number of iterations is reached to generate a sample sequence comprises: Based on the proposed distribution, the acceptance probability is calculated; the acceptance probability represents the ratio of the probability of the current state to the candidate state and the reverse process thereof; A value is randomly generated in the interval [0, 1]. If the reception probability is not less than the randomly generated value, the candidate state is used as the current state of the next iteration; otherwise, the current state is maintained for the next iteration; The current state of each iteration is obtained, and the sample sequence is generated based on the current state.

5. The method according to claim 1, characterized in that Obtaining a first structured query verification statement according to the parameter value includes: The parameter values ​​are concatenated to obtain the first structured query verification statement.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a second structured query verification statement according to the boundary value of the parameter type value range; The second structured query verification statement is sent to the target database end for verification to obtain a verification result.

7. The method according to claim 6, characterized in that Obtaining a second structured query verification statement according to the boundary value of the parameter type value range includes: The boundary values ​​of the parameter type value range are concatenated to obtain a second structured verification statement.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the parameter list is a constant, the constants are concatenated to obtain a third structured query verification statement.

9. A data consistency verification device, characterized in that: The device comprises: An acquisition module, used for acquiring a built-in function of a data object of a source database, and obtaining a parameter list of the built-in function according to the built-in function; A sampling module, used to obtain a parameter type value range of the parameter list according to the parameter value type of the parameter list; and to obtain a parameter value within the parameter type value range using a random sampling algorithm according to the parameter type value range; The verification module is used to obtain a first structured query verification statement according to the parameter value; send the first structured query verification statement to the target database end for verification, and obtain a verification result.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images