Data checking method and device, electronic equipment and storage medium

Abstract

The invention discloses a data checking method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a data mapping relation according to source system metadata, target system metadata and a data extraction condition, and the data mapping relation represents a relation when data is migrated from a source system or reconstructed to a target system; and determining data export configuration of the source system and data import configuration of the target system according to the data mapping relationship. Second data are determined according to the first data, the second data are processed according to the data export configuration and the data import configuration in sequence, third data and fourth data are obtained, the first data are to-be-compared data of the target system, the second data are comparison data of the first data in the source system, and the third data and the first data are isomorphic. And determining a check result according to the data mapping relationship, the first data and the fourth data. According to the method, unified configurable data checking processing is realized, so that configuration data in each step is conveniently and correctly generated, and the workload and the maintenance cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data verification method, device, electronic equipment and storage medium. Background Art

[0002] During system migration or reconstruction, developers must not only ensure the accuracy of system functionality but also verify its accuracy. Therefore, developers will develop export and import code based on specific data verification requirements. These code will be used to import data from the old system into the new system. Developers will then use specially developed verification code to verify the imported data from the old system against the data to be verified, and will use specially developed presentation code to display the verification results.

[0003] However, since independent programs are required in the four stages of data import, data export, data verification, and data display, multiple sets of programs need to be developed for different verification scenarios, resulting in a large amount of repetitive development work and high development and maintenance costs. Summary of the Invention

[0004] The present invention provides a data verification method, device, electronic device and storage medium, which solves the current problem of needing to use independent programs for different steps, resulting in a large amount of repeated development work and high development and maintenance costs. It realizes unified and configurable data verification processing, so that the configuration data of each step can be generated conveniently and accurately, reducing the workload of program development and maintenance costs.

[0005] According to one aspect of the present invention, a data verification method is provided, the method comprising:

[0006] The data mapping relationship is determined based on the metadata of the source system, the metadata of the target system, and pre-set data extraction conditions, wherein the data mapping relationship is used to represent the mapping relationship when data is migrated or reconstructed from the source system to the target system.

[0007] Determine the data export configuration of the source system and the data import configuration of the target system based on the data mapping relationship.

[0008] The second data is determined based on the first data, and the second data is processed in sequence according to the data export configuration and the data import configuration to obtain third data and fourth data, wherein the first data is the data to be compared in the target system, the second data is the comparison data of the first data in the source system, and the third data and the first data are isomorphic data.

[0009] A verification result is determined according to the data mapping relationship, the first data, and the fourth data.

[0010] According to another aspect of the present invention, there is provided a data checking device, comprising:

[0011] The relationship determination module is used to determine the data mapping relationship based on the metadata of the source system, the metadata of the target system and pre-set data extraction conditions, wherein the data mapping relationship is used to represent the mapping relationship when data is migrated or reconstructed from the source system to the target system.

[0012] The configuration determination module is used to determine the data export configuration of the source system and the data import configuration of the target system according to the data mapping relationship.

[0013] A processing module is used to determine second data based on first data, and process the second data in turn according to the data export configuration and the data import configuration to obtain third data and fourth data, wherein the first data is the data to be compared in the target system, the second data is the comparison data of the first data in the source system, and the third data and the first data are isomorphic data.

[0014] The verification module is used to determine a verification result according to the data mapping relationship, the first data and the fourth data.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the data verification method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data verification method according to any embodiment of the present invention when executed.

[0020] The data verification method provided by the embodiment of the present invention determines the data mapping relationship according to the metadata of the source system, the metadata of the target system and the pre-set data extraction conditions; determines the data export configuration of the source system and the data import configuration of the target system respectively according to the data mapping relationship; determines the second data according to the first data, and processes the second data according to the data export configuration and the data import configuration in turn to obtain third data and fourth data; and determines the verification result according to the data mapping relationship, the first data and the fourth data. The above technical solution, on the one hand, directly determines the data mapping relationship used to characterize the mapping relationship when the data is migrated or reconstructed from the source system to the target system according to the metadata, the metadata of the target system and the data extraction conditions required for this data verification, and can realize the interaction and integration relationship between the source system and the target system according to the metadata, which is convenient for quickly determining the changes in the data structure or data when implementing data import, data export and other two steps from the source system or the target system, and provides a unified configuration standard for data verification. At the same time, it realizes the clear identification of the data to be verified and the data associated with it according to the data extraction conditions, which is convenient for accurately and quickly selecting the data to be verified and the data associated with it from the batch data. On the other hand, by directly determining the data export configuration of the source system and the data import configuration of the target system based on the data mapping relationship, it is possible to quickly and accurately configure the data configuration information for the data export and data import steps in advance. Finally, based on the data mapping relationship, the configurations of each step that are related to each other are determined, and the data processing of each step is completed according to the configuration, which solves the current problem of needing to use independent programs for different steps, resulting in a large amount of repeated development work and high development and maintenance costs. It realizes unified and configurable data export, data import, and data verification processing, so that the configuration data of each step can be generated conveniently and accurately, reducing the workload of program development and maintenance costs. At the same time, the required configuration can be provided for each step, and there is a correlation between the configurations of each step, which solves the problem that the current use of a common program will cause the configurations of each step to be separated, resulting in mismatched configurations between different steps, and thus leading to confusion of column information of the data to be compared.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flowchart of a data verification method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a flow chart of another data verification method provided by an embodiment of the present invention;

[0025] Figure 3 A schematic structural diagram of a data verification device provided by an embodiment of the present invention;

[0026] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "source", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0029] Figure 1This is a flow chart of a data verification method provided by an embodiment of the present invention. This embodiment is applicable to the case of verifying data during system migration and reconstruction. The method can be executed by a data verification device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. In this embodiment, the electronic device can be an electronic device equipped with an active system or a target system. Figure 1 As shown, the method includes:

[0030] S101 : Determine a data mapping relationship based on metadata of a source system, metadata of a target system, and pre-set data extraction conditions.

[0031] Among them, the source system is the old system before the system is migrated or reconstructed. The target system is the new system after the system is migrated or reconstructed. In this embodiment, the source system and the target system are respectively installed on different electronic devices, but the data verification device in this embodiment can be configured on the electronic device installed on the source system or the electronic device installed on the target system. For the convenience of the following description, this embodiment will configure the data verification device on the electronic device installed on the target system. Metadata is data that describes the data attributes, structure, relationship and context information in the system. The data extraction condition is the condition that in this data verification, the data that needs to be verified and the data associated with the data that needs to be verified need to be selected from a large amount of data. The data mapping relationship is used to characterize the mapping relationship when data is migrated or reconstructed from the source system to the target system.

[0032] Specifically, source system metadata is extracted from the source system's database, and target system metadata is extracted from the target system's database. Furthermore, developers determine data extraction criteria based on the data to be verified within the large volume of data and its associated data. Based on the source system metadata, target system metadata, and data extraction criteria, they determine the data mapping relationship for this verification. This data configuration relationship provides the basis for determining corresponding configurations and configuration information in various subsequent data verification steps.

[0033] In this embodiment, the data mapping relationship used to characterize the mapping relationship when migrating or reconstructing data from the source system to the target system is determined directly based on the metadata of the source system, the metadata of the target system, and the data extraction conditions required for this data verification. This allows the interaction and integration relationship between the source and target systems to be determined based on the metadata, facilitating the rapid determination of data structure or data changes during the subsequent data import, data export, and other steps from the source or target system, thereby providing a unified configuration standard for data verification. Furthermore, the data that needs to be verified and the data associated with it are clearly identified based on the data extraction conditions, facilitating the accurate and rapid selection of the data required for verification and the data associated with it from batch data.

[0034] S102: Determine the data export configuration of the source system and the data import configuration of the target system according to the data mapping relationship.

[0035] In this embodiment, the target system is used to identify data with similar structures in the source system. The data is then exported and imported into the target system. Data export configuration specifies the rules and requirements that must be followed when exporting data from the source system, while data import configuration specifies the rules and requirements that must be followed when importing data into the target system.

[0036] Specifically, from the data mapping relationship, the table definition and column definition describing the data format in the relational database of the source system are determined, and this is used as the data export configuration. Also, from the data mapping relationship, the table definition and column definition describing the data format in the relational database of the target system are determined, and this is used as the data import configuration.

[0037] In this embodiment, the data export configuration of the source system and the data import configuration of the target system are determined directly according to the data mapping relationship, so that the data configuration information can be quickly and accurately configured in advance for the data export and data import steps, providing a basis for subsequently exporting data using the data export configuration and importing data using the data import configuration.

[0038] S103: Determine second data according to the first data, and process the second data in sequence according to the data export configuration and the data import configuration to obtain third data and fourth data.

[0039] The first data is the target system's data to be compared, the second data is the comparison data for the first data in the source system, and the third data is isomorphic to the first data. The third data is the data after export, and the fourth data is the data after import. In this embodiment, the first data is the data generated after the target system is running, and the second data is the data generated after the source system is running. Furthermore, the data newly input into the target and source systems before and during operation is consistent.

[0040] Specifically, based on the data to be verified, i.e., the first data, data with a similar structure to the first data can be identified from the source system. This data serves as the comparison data for the first data, i.e., the second data. In this case, the second data can be exported from the source system using the data export configuration to obtain the third data. The third data can then be imported into the target system using the data import configuration to obtain the fourth data.

[0041] In this embodiment, the second data is processed sequentially using the data export configuration and data import configuration determined based on the data mapping relationship to obtain the third data after the data is exported and the fourth data after the data is imported. This allows a data mapping relationship to be quickly applied to the data export and data import steps, making the data mapping relationship highly reusable and extensible, enabling it to be quickly and flexibly applied to the data export and data import stages. Furthermore, the data mapping relationship enables unified and configurable data export and data import, allowing the configuration data for data export and data import to be generated conveniently and accurately.

[0042] S104: Determine a verification result according to the data mapping relationship, the first data, and the fourth data.

[0043] Specifically, since the first data is the data to be checked this time, and the fourth data is the data after data export and data import from the source system, the first data and the fourth data can be compared and checked, and the accuracy of the data migration and the accuracy of the processing results can be determined based on the verification results. Therefore, according to the data mapping relationship, the table and column name information in the table definition and column definition that describe the data format in the relational database of the target system can be used to generate a data verification configuration. The data verification configuration is used to provide accurate verification criteria for checking the first data and the fourth data one by one, and is used to standardize the format of the verification results. Based on the data mapping relationship, the data in the first data and the fourth data are checked one by one to obtain a comparison result, and the comparison result is organized into a business-recognizable verification result according to business needs.

[0044] In this embodiment, the data verification configuration can still be determined based on the data mapping relationship, and the data verification of the first data and the fourth data can be achieved based on the data verification configuration. This solves the current problem of needing to use independent programs for different steps, resulting in a large amount of repeated development work and high development and maintenance costs. It realizes unified and configurable data export, data import, and data verification processing, so that the configuration data of each step can be generated conveniently and accurately, reducing the workload and maintenance costs of program development. At the same time, the required configuration can be provided for each step, and the configuration of each step is related. This solves the problem that the current use of a common program will cause the configuration of each step to be disconnected, resulting in mismatched configurations between different steps, and thus leading to confusion of column information of the data to be compared.

[0045] The data verification method provided by the embodiment of the present invention determines the data mapping relationship according to the metadata of the source system, the metadata of the target system and the pre-set data extraction conditions; determines the data export configuration of the source system and the data import configuration of the target system respectively according to the data mapping relationship; determines the second data according to the first data, and processes the second data according to the data export configuration and the data import configuration in turn to obtain third data and fourth data; and determines the verification result according to the data mapping relationship, the first data and the fourth data. The above technical solution, on the one hand, directly determines the data mapping relationship used to characterize the mapping relationship when the data is migrated or reconstructed from the source system to the target system according to the metadata, the metadata of the target system and the data extraction conditions required for this data verification, and can realize the interaction and integration relationship between the source system and the target system according to the metadata, which is convenient for quickly determining the changes in the data structure or data when implementing data import, data export and other two steps from the source system or the target system, and provides a unified configuration standard for data verification. At the same time, it realizes the clear identification of the data to be verified and the data associated with it according to the data extraction conditions, which is convenient for accurately and quickly selecting the data to be verified and the data associated with it from the batch data. On the other hand, by directly determining the data export configuration of the source system and the data import configuration of the target system based on the data mapping relationship, it is possible to quickly and accurately configure the data configuration information for the data export and data import steps in advance. Finally, based on the data mapping relationship, the configurations of each step that are related to each other are determined, and the data processing of each step is completed according to the configuration, which solves the current problem of needing to use independent programs for different steps, resulting in a large amount of repeated development work and high development and maintenance costs. It realizes unified and configurable data export, data import, and data verification processing, so that the configuration data of each step can be generated conveniently and accurately, reducing the workload of program development and maintenance costs. At the same time, the required configuration can be provided for each step, and there is a correlation between the configurations of each step, which solves the problem that the current use of a common program will cause the configurations of each step to be separated, resulting in mismatched configurations between different steps, and thus leading to confusion of column information of the data to be compared.

[0046] Figure 2 This is a flow chart of another data verification method provided by an embodiment of the present invention. Based on the above embodiment, this embodiment describes in detail how to determine the data mapping relationship, determine the configuration of each step, and process the data according to the configuration of each step to obtain the verification result. Figure 2 As shown, the method includes:

[0047] S201 : Determine the field names of the source system according to the metadata of the source system, and determine the field names of the target system according to the metadata of the target system.

[0048] The field name is the name of the data item in the database used to identify the system.

[0049] Specifically, metadata is extracted from the databases of the source system and the target system respectively, and the field names of the system and information related to the field names, such as field attribute information, are determined from the metadata.

[0050] S202: Determine first field attribute information according to the extraction filter condition, the field name of the target system, and the field name of the source system.

[0051] The data extraction conditions include extraction filtering conditions. In this embodiment, the extraction filtering conditions are used to filter out information such as data field names that do not need to be checked or are unimportant.

[0052] Specifically, the field names of the target system and the field names of the source system may be screened and filtered using the extraction filter condition, and the required field data information may be selected as the first field attribute information.

[0053] S203: Determine the second field attribute information according to the extraction association condition, the field name of the target system, and the field name of the source system.

[0054] The data extraction conditions also include extraction association conditions. In this embodiment, the extraction association conditions are used to select information such as data field names that are associated with the data that needs to be checked or important data.

[0055] Specifically, according to the extraction association condition, the field name of the target system and the field name of the source system can be associated with each other to determine information, and the field data information associated with the data that needs to be checked or important data this time can be selected as the second field attribute information.

[0056] It is worth noting that S202 and S203 are parallel steps and can be executed simultaneously or sequentially.

[0057] S204: Determine a mapping relationship of the configuration information according to the first field attribute information and the second field attribute information.

[0058] Specifically, the first field attribute information and the second field attribute information are combined to obtain a mapping relationship of the configuration information.

[0059] For example, the mapping relationship of the configuration information is shown in a table as follows:

[0060]

[0061] Among them, the primary key is used to uniquely identify the field or field combination of a record to ensure the uniqueness and integrity of the data. The field description is used to provide a textual description of the meaning of the field. The data type is the data format that the field is allowed to store, such as the "fixed-length character (8) CHAR (8)" format and the "date type data" format. The length is the maximum length of the storage content for fields of types such as strings and numbers. Nullable indicates whether the field is allowed to store null values. The default value indicates the default value automatically filled by the system when inserting data into the table if the value of the field is not explicitly specified. The source field name is the field name in the upstream system or data source corresponding to the current field, which in this embodiment is the field name in the source system database. The source system nullable means that when the target system is not nullable but the source system is nullable, it needs to be marked. At this time, the null value of the source system (marked as null in the system) needs to be converted to a null value. In this embodiment, since the "null value" and "empty value" marking formats of the source system and target system databases are different, if the conversion is not done, data containing "null value" cannot be imported into the target system.

[0062] In this embodiment, based on the correlation between the source system and the target system when processing data, that is, the object processed by the data import step is the export result of the data export step, the object processed by the data verification step is the import result of the data import step, and the object processed by the data verification and display step is the verification result of the data verification step, the field names, extraction filtering conditions and extraction association conditions of the source system and the target system can be used to determine a unified data mapping relationship. This data mapping relationship can generate a unified general configuration for subsequent steps, thereby realizing that the configuration information of each step is uniformly generated by the mapping relationship.

[0063] S205: Determine a data export configuration carrying a query statement according to the mapping relationship and metadata in the source system.

[0064] The query statement is a Structured Query Language (SQL) statement.

[0065] Specifically, according to the mapping relationship, the column definition, i.e., conversion information, table, data filtering conditions, and other information are determined from the table definition and column definition that describe the data format in the relational database of the source system, assembled into an SQL statement, and the assembled SQL statement is used as the data export configuration.

[0066] S206: Determine a query statement carrying a verification rule according to the mapping relationship and the metadata in the target system, and determine a data import configuration according to the query statement carrying the verification rule.

[0067] Specifically, according to the mapping relationship, information such as column definition and table name is determined from the table definition and column definition that describe the data format in the relational database of the target system as the data import configuration.

[0068] Exemplarily, the determined information may include: the target table, the number of file columns, the type of each column (such as date type date, character type char, decimal type decimal, and timestamp type timestamp), and the verification rules for each column (such as the default value, whether to allow nulls, date validity and format, string length verification, and timestamp format).

[0069] In this embodiment, since the data export configuration and the data import configuration are both determined based on the mapping relationship, the attribute order of the table definition and column definition describing the data format in the relational database of the source system and the target system is consistent, thereby ensuring the consistency of the data exported according to the data export configuration and the data import configuration.

[0070] S207: Determine second data based on the first data.

[0071] Specifically, after determining the first data in the target system that needs to be checked this time, the field name can be used to determine the data in the source system that has the highest similarity to the field name of the first data in the target system as the second data.

[0072] For example, a list of fields is extracted from the first data, and data containing similar fields is searched in the source system. The number of fields extracted from the first data and the data containing similar fields in the source system that are completely identical or semantically similar is calculated, with a focus on data with numeric, string, and date data types. With primary keys and index fields as the focus, the team determines whether the primary key fields are identical or semantically equivalent, and whether the index fields are consistent. Data samples are then extracted for verification and similarity evaluation, ultimately determining the second data. Manual verification can also be performed.

[0073] S208: Export the second data according to the data export configuration carrying the query statement to obtain third data that is isomorphic to the first data.

[0074] Specifically, the second data can be exported using the data export configuration to obtain a comparison file exported from the source system that is isomorphic to the first data, i.e., the table to be compared, and used as the third data.

[0075] S209: Import the third data into the target system according to the data import configuration to obtain fourth data.

[0076] Specifically, the third data is imported into the target system according to the data import configuration and into a comparison table isomorphic with the first data, thereby obtaining the fourth data.

[0077] S210: Determine data verification elements corresponding to the data to be verified in the first data and the fourth data according to the verification configuration in the mapping relationship.

[0078] Among them, data verification elements are key dimensional elements that need to be paid attention to during the data verification process.

[0079] Specifically, because data representations vary across systems, it is necessary to first determine the data verification elements for the data to be verified based on the verification configuration in the mapping relationship. In this embodiment, the data verification elements include error verification elements, quality verification elements, and rule verification elements. Error verification elements primarily include rounding errors and calculation errors; quality verification elements primarily include data integrity and accuracy; and rule verification elements primarily include data consistency and logical rationality.

[0080] In this embodiment, since the data verification elements for the data to be verified can be determined based on the mapping relationship, the accuracy and completeness of the data verification can be achieved. Furthermore, since the mapping relationship is determined based on metadata in the source and target systems, it can be completely decoupled from the specific verification objects, realizing a universal batch data verification method.

[0081] S211. Determine comparison information based on the data verification element, the first data, and the fourth data.

[0082] Specifically, there are two methods for comparing the first data with the fourth data using the data verification elements, and either or both of these methods may be performed.

[0083] In one implementation, the number of records in the first data and the fourth data is determined, and the first data and the fourth data are checked one by one according to the data checking elements corresponding to each piece of data.

[0084] Specifically, each data item can be presented in a table format. Therefore, the number of records in the first and fourth data items can be determined first. The number of records is used to determine the number of data items that need to be checked. Afterwards, each data item that needs to be checked is checked item by item. During the checking process, the data needs to be checked based on the data checking elements.

[0085] Another implementation method is to extract preset data from the first data and the fourth data, and verify the preset data in the first data with the preset data in the fourth data according to the data verification elements corresponding to the preset data.

[0086] Specifically, when the amount of data is large and this verification is only a sampling verification, this method can be selected to select important data or data that needs to be verified this time from the first data and the fourth data as preset data, and verify the preset data in the first data and the preset data in the fourth data based on the data verification elements.

[0087] After completing the data verification according to the above two implementation methods, the verification information corresponding to the data with differences can be determined as the comparison information.

[0088] Optionally, after obtaining the comparison information, a standard storage method for the comparison information including the verification information can be determined based on the mapping relationship, and the comparison information can be stored according to the standard storage method to obtain a standard result. That is, after obtaining the comparison information, the comparison information, the comparison result, and the description of the comparison information can be uniformly stored in the verification result table according to the format in the mapping relationship, that is, according to the standard format.

[0089] S212: Determine and display the verification result based on the mapping relationship and the comparison information.

[0090] Specifically, after obtaining the comparison information, it is generally stored locally or in the cloud. When staff need to view the verification results for a certain data, they can retrieve the verification results corresponding to that data from the local or cloud. Therefore, after obtaining the comparison information, the verification results can be generated based on the comparison data and mapping relationships corresponding to the comparison information according to the current business needs. Furthermore, after the verification results are generated, they can be displayed to the user.

[0091] For example, after obtaining the comparison information, the comparison information, the comparison results, and the description of the comparison information are uniformly stored in a standardized format in a verification result table. Therefore, before displaying the verification results, a parsing method can be determined based on the mapping relationship, and the standardized results can be parsed to obtain the comparison information. The parsing method for the standardized results can be determined based on the mapping relationship. Therefore, the parsing method can be determined based on the mapping relationship, and the standardized results can be parsed. Furthermore, based on the comparison information, the verification information corresponding to the data where there is a difference between the first data and the fourth data is spliced ​​to determine and display the verification results.

[0092] In this embodiment, since the comparison information and verification results are determined based on the mapping relationship, it is possible to determine the configuration of each step according to the mapping relationship and perform data processing on each step in different verification processes as long as the source system and the target system remain unchanged, thereby achieving a fast and flexible response to the verification needs of different verification objects with low repetitive development workload.

[0093] Figure 3This is a structural diagram of a data verification device provided by an embodiment of the present invention. Figure 3 As shown, the device includes:

[0094] The relationship determination module 301 is used to determine a data mapping relationship based on the metadata of the source system, the metadata of the target system, and pre-set data extraction conditions. The data mapping relationship is used to represent the mapping relationship when data is migrated or reconstructed from the source system to the target system;

[0095] Configuration determination module 302, used to determine the data export configuration of the source system and the data import configuration of the target system according to the data mapping relationship;

[0096] Processing module 303 is configured to determine second data based on the first data, and process the second data in accordance with the data export configuration and the data import configuration to obtain third data and fourth data, wherein the first data is the data to be compared in the target system, the second data is the comparison data of the first data in the source system, and the third data is isomorphic with the first data;

[0097] The checking module 304 is configured to determine a checking result according to the data mapping relationship, the first data, and the fourth data.

[0098] Optionally, the data extraction conditions include extraction filtering conditions and extraction association conditions; the relationship determination module 301 is specifically used to:

[0099] Determine the field name of the source system according to the metadata of the source system, and determine the field name of the target system according to the metadata of the target system; determine the first field attribute information according to the extraction filtering condition, the field name of the target system and the field name of the source system; determine the second field attribute information according to the extraction association condition, the field name of the target system and the field name of the source system; determine the mapping relationship of the configuration information according to the first field attribute information and the second field attribute information.

[0100] Optionally, the configuration determination module 302 is specifically configured to:

[0101] The data export configuration including the query statement is determined based on the mapping relationship and the metadata in the source system. The query statement including the validation rules is determined based on the mapping relationship and the metadata in the target system, and the data import configuration is determined based on the query statement including the validation rules.

[0102] Optionally, the processing module 303 is specifically configured to:

[0103] The second data is exported according to the data export configuration carrying the query statement to obtain third data that is isomorphic to the first data; and the third data is imported into the target system according to the data import configuration to obtain fourth data.

[0104] Optionally, the checking module 304 is specifically configured to:

[0105] According to the verification configuration in the mapping relationship, determine the data verification elements corresponding to the data to be verified in the first data and the fourth data; determine the comparison information based on the data verification elements, the first data and the fourth data; determine and display the verification results based on the mapping relationship and the comparison information.

[0106] Optionally, the comparison information is determined based on the data verification element, the first data, and the fourth data. The verification module 304 is specifically configured to:

[0107] Determine the number of records in the first data and the fourth data, and perform data verification on the first data and the fourth data one by one according to the data verification element corresponding to each data, and / or extract preset data from the first data and the fourth data, and verify the preset data in the first data with the preset data in the fourth data according to the data verification element corresponding to the preset data; determine the verification information corresponding to the data with differences in the verification as comparison information.

[0108] Optionally, after determining the verification information corresponding to the data with differences as the comparison information, the verification module 304 is further configured to:

[0109] Determine the standard storage mode of the comparison information including the verification information according to the mapping relationship, and store the comparison information according to the standard storage mode to obtain a standard result; determine and display the verification result according to the mapping relationship and the comparison information. The verification module 304 is specifically used to:

[0110] Determine the parsing method according to the mapping relationship, and parse the format result to obtain comparison information; based on the comparison information, splice the verification information corresponding to the data where there is a difference between the first data and the fourth data, and determine and display the verification result.

[0111] The data checking device provided in the embodiment of the present invention can execute the data checking method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0112] Figure 4Schematic diagram of the structure of an electronic device provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0113] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data verification method.

[0116] In some embodiments, the data verification method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data verification method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data verification method in any other appropriate manner (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0123] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0124] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data verification method, characterized in that: The method comprises: Determining a data mapping relationship based on metadata of a source system, metadata of a target system, and pre-set data extraction conditions, wherein the data mapping relationship is used to represent a mapping relationship when data is migrated or reconstructed from the source system to the target system; Determine the data export configuration of the source system and the data import configuration of the target system respectively according to the data mapping relationship; Determining second data based on the first data, and processing the second data in sequence according to the data export configuration and the data import configuration to obtain third data and fourth data, wherein the first data is the data to be compared in the target system, the second data is comparison data of the first data in the source system, and the third data and the first data are isomorphic data; A verification result is determined according to the data mapping relationship, the first data, and the fourth data.

2. The data verification method according to claim 1, characterized in that: The data extraction conditions include extraction filtering conditions and extraction association conditions; Determining the data mapping relationship based on the metadata of the source system, the metadata of the target system, and pre-set data extraction conditions includes: Determining a field name of a source system according to the metadata of the source system, and determining a field name of a target system according to the metadata of the target system; Determining first field attribute information according to the extraction filter condition, the field name of the target system, and the field name of the source system; determining second field attribute information according to the extraction association condition, the field name of the target system, and the field name of the source system; A mapping relationship between the configuration information is determined according to the first field attribute information and the second field attribute information.

3. The data verification method according to claim 1, wherein: The determining, according to the data mapping relationship, respectively the data export configuration of the source system and the data import configuration of the target system includes: Determining a data export configuration carrying a query statement according to the mapping relationship and metadata in the source system; A query statement carrying a verification rule is determined according to the mapping relationship and metadata in the target system, and a data import configuration is determined according to the query statement carrying the verification rule.

4. The data checking method according to claim 3, characterized in that: The step of sequentially processing the second data according to the data export configuration and the data import configuration to obtain third data and fourth data includes: Exporting the second data according to the data export configuration carrying the query statement to obtain the third data having the same structure as the first data; The third data is imported into the target system according to the data import configuration to obtain the fourth data.

5. The data checking method according to claim 1, wherein: Determining a verification result according to the data mapping relationship, the first data, and the fourth data includes: Determining, according to the verification configuration in the mapping relationship, data verification elements corresponding to the data to be verified in the first data and the fourth data; determining comparison information based on the data verification element, the first data, and the fourth data; The verification result is determined and displayed based on the mapping relationship and the comparison information.

6. The data checking method according to claim 5, characterized in that: The determining of comparison information based on the data verification element, the first data, and the fourth data includes: Determining the number of records in the first data and the fourth data, and performing a data check on the first data and the fourth data one by one according to a data check element corresponding to each piece of data, and / or extracting preset data from the first data and the fourth data, and checking the preset data in the first data with the preset data in the fourth data according to the data check element corresponding to the preset data; The verification information corresponding to the data with differences is determined as the comparison information.

7. The data checking method according to claim 6, characterized in that: After determining the verification information corresponding to the data with differences as comparison information, the following steps are also included: Determining a standard storage mode for comparison information including the verification information according to the mapping relationship, and storing the comparison information according to the standard storage mode to obtain a standard result; The determining and displaying the verification result according to the mapping relationship and the comparison information includes: Determine a parsing method according to the mapping relationship, and parse the format result to obtain comparison information; According to the comparison information, the verification information corresponding to the data in which there is a difference between the first data and the fourth data is spliced, and the verification result is determined and displayed.

8. A data checking device, characterized in that: The device comprises: a relationship determination module, configured to determine a data mapping relationship based on metadata of a source system, metadata of a target system, and pre-set data extraction conditions, wherein the data mapping relationship is used to represent a mapping relationship when data is migrated or reconstructed from the source system to the target system; a configuration determination module, configured to determine the data export configuration of the source system and the data import configuration of the target system respectively according to the data mapping relationship; a processing module, configured to determine second data based on the first data, and sequentially process the second data according to the data export configuration and the data import configuration to obtain third data and fourth data, wherein the first data is the data to be compared in the target system, the second data is comparison data of the first data in the source system, and the third data and the first data are isomorphic data; A verification module is used to determine a verification result according to the data mapping relationship, the first data and the fourth data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data checking method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data verification method according to any one of claims 1 to 7 when executed.

Images