Data checking method and related equipment

Abstract

The invention provides a data checking method and related equipment. The method comprises the following steps: extracting first data from data contained in a source end system, wherein the first data comprises a plurality of fields; determining a primary key field in the plurality of fields contained in the first data, and obtaining a primary key value in the primary key field; searching second data related to the primary key value from data contained in a target end system; wherein at least part of the data in the target end system is the data synchronized from the source end system, and the at least part of the data comprises the second data; and checking the first data and the second data, and executing corresponding data management on the target end system based on a checking result.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of data management technology, and in particular, to a data verification method and related equipment. Background Art

[0002] ETL (Extract, Transform, Load) is a data integration technology that is mainly used to extract data from the source system, transform the extracted data to adapt it to the format or requirements of the target system, and finally load the processed data into the target system. ETL technology plays an important role in modern data management. It not only helps enterprises and organizations integrate and process data from different sources, but also provides a solid foundation for advanced analysis and decision support. Whether it is building a data warehouse, managing a data lake, or realizing data synchronization and migration, ETL is an indispensable technical means.

[0003] However, in traditional ETL technology, complex data conversion rules and logic may lead to inconsistent data changes, which in turn affects the accuracy of the final data integration results. For example, during the data conversion process (which may include a series of complex processes such as data normalization or merging), technical defects or human errors may lead to the loss of part or even all of the data, resulting in incomplete and inaccurate data in the target system, making it impossible to keep consistent with the data in the source system. These data quality issues not only affect corporate decision-making, but may also bring serious compliance risks.

[0004] Therefore, how to ensure data consistency between the target system and the source system based on ETL technology is an urgent problem to be solved. Summary of the invention

[0005] In view of this, one or more embodiments of the present specification provide a data verification method and related devices.

[0006] In a first aspect, this specification provides a data verification method, the method comprising:

[0007] Extracting first data from data contained in a source end system, wherein the first data includes a plurality of fields;

[0008] Determine a primary key field among the multiple fields included in the first data, and obtain a primary key value in the primary key field;

[0009] Finding second data related to the primary key value from the data contained in the target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data includes the second data;

[0010] The first data and the second data are checked, and corresponding data management is performed on the target end system based on the checking result.

[0011] In a second aspect, this specification provides a data verification device, the device comprising:

[0012] A source-end data extraction unit, configured to extract first data from data contained in a source-end system, wherein the first data includes a plurality of fields;

[0013] a primary key value acquiring unit, configured to determine a primary key field among the multiple fields included in the first data, and acquire a primary key value in the primary key field;

[0014] A target-side data extraction unit, configured to search for second data related to the primary key value from data contained in the target-side system; wherein at least part of the data in the target-side system is data synchronized from the source-side system, and the at least part of the data includes the second data;

[0015] A data checking unit is used to check the first data and the second data, and perform corresponding data management on the target end system based on the checking result.

[0016] Accordingly, the present specification also provides a computing device, comprising: a memory and a processor; the memory stores a computer program / instruction that can be executed by the processor; when the processor executes the computer program / instruction, it executes the data verification method described in the first aspect above.

[0017] Accordingly, the present specification also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the data verification method as described in the first aspect above is executed.

[0018] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the data verification method described in the first aspect above is performed.

[0019] In summary, after extracting the first data from the source end system, the present application can determine the primary key field among the multiple fields contained in the first data, and obtain the primary key value in the primary key field, and then reversely query the target end system based on the primary key value to find the second data related to the primary key value from the target end system. Finally, check the first data and the second data to verify the data consistency between the target end system and the source end system. In this way, the present application can verify the data consistency between the target end system and the source end system on the basis of performing data integration from the source end system to the target end system (for example, synchronizing the data in the source end system to the target end system) by adding additional data verification steps, thereby ensuring the data integrity and accuracy of the target end system, so as to provide users with more reliable and secure data services. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0021] Figure 2 is another system architecture schematic diagram provided by an exemplary embodiment;

[0022] Figure 3 is a structural schematic diagram of a data batch component provided by an exemplary embodiment;

[0023] Figure 4 It is a flowchart of a data review method provided by an exemplary embodiment;

[0024] Figure 5 is a flow chart of a data checking method provided by an exemplary embodiment;

[0025] Figure 6 is a structural schematic diagram of a data checking device provided by an exemplary embodiment;

[0026] Figure 7 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0030] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] ETL is a data integration technology that is widely used in various scenarios such as data warehouses and big data processing processes. It includes three core steps:

[0032] Extract: Extract raw data from one or more source systems. The source system can be any possible data source, such as a relational database, a non-relational database, a CSV file, an XML file, an API interface, etc. This specification does not specifically limit this.

[0033] Transform: The raw data extracted from the source system is cleaned, converted, and normalized to adapt to the format and requirements of the target system. This step may include filling in default values, extracting qualified data, removing duplicate records, data type conversion, data aggregation, and data quality checks, which are not specifically limited in this manual.

[0034] Load: Load the transformed data (i.e. the result of Transform) into the target system, which can also be loaded into a data warehouse, data lake, or any other possible system for subsequent analysis and report generation. The target system can also be a relational database, a non-relational database, a CSV file, an XML file, or an API interface, etc. This specification does not specifically limit this.

[0035] With the increase in service demand and data complexity, ETL will involve more complex data transformation rules and logic, which are likely to lead to inconsistent data changes, thus affecting the accuracy of the final data integration results. For example, during the transformation process, technical defects or human errors are likely to cause the loss of part or even all of the data, resulting in incomplete and inaccurate data in the target system, making it impossible to keep consistent with the data in the source system. These data quality issues not only affect corporate decision-making, but may also bring serious compliance risks.

[0036] Based on this, this specification provides a technical solution. Through the additional data verification step, the data consistency between the target system and the source system can be verified on the basis of data integration, so as to provide users with more reliable and secure data services.

[0037] During implementation, first data may be extracted from the data contained in the source end system, and the first data may include multiple fields. Then, a primary key field among the multiple fields contained in the first data is determined, and the primary key value in the primary key field is obtained. Furthermore, second data related to the primary key value is found from the data contained in the target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data includes the second data. Finally, the first data and the second data may be checked, and corresponding data management may be performed on the target end system based on the check result.

[0038] In the above technical scheme, after extracting the first data from the source end system, the present application can determine the primary key field among the multiple fields contained in the first data, and obtain the primary key value in the primary key field, and then reversely query the target end system based on the primary key value to find the second data related to the primary key value from the target end system. Finally, check the first data and the second data to verify the data consistency between the target end system and the source end system. In this way, the present application can verify the data consistency between the target end system and the source end system on the basis of performing data integration from the source end system to the target end system (for example, synchronizing the data in the source end system to the target end system) by adding additional data verification steps, thereby ensuring the data integrity and accuracy of the target end system, so as to provide users with more reliable and secure data services.

[0039] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 1 The system architecture shown in the figure or a similar system architecture is specifically implemented. Figure 1As shown, the system architecture may include a source end system 100, a target end system 200 and a data checking device 300. The source end system 100, the target end system 200 and the data checking device 300 may establish a communication connection through any possible means, and this specification does not specifically limit this.

[0040] The source end system 100 and the target end system 200 may be two end systems in data integration, that is, at least part of the data in the target end system 200 is data synchronized from the source end system 100 .

[0041] It should be noted that this specification does not specifically limit the specific scenarios of data integration.

[0042] In an illustrated embodiment, the scenario of data integration may be a scenario of master-slave database synchronization, wherein the source end system 100 may be a master database in a master-slave database system, and the target end system 200 may be a slave database in a master-slave database system, and the data in the master database may be synchronized to the slave database in real time or periodically to implement data backup, ensuring that when an unrecoverable failure occurs in the master database, the service may be quickly switched to the slave database, thereby continuing to provide services to users and ensuring that the user experience is not affected.

[0043] In an embodiment shown, the scenario of data integration can be a scenario of data warehouse or data lake synchronization. The source end system 100 can be various data sources (such as sales systems or inventory systems of various enterprises), and the target end system 200 can be a data warehouse or data lake. Data is extracted from various data sources and synchronized to the data warehouse or data lake, which can be used for analysis and report generation, etc. This specification does not make specific limitations on this.

[0044] like Figure 3 As shown, in an embodiment shown, the data checking device 300 may first extract first data from the data contained in the source end system 100, and the first data may include multiple fields. Further, the data checking device 300 may determine a primary key field among the multiple fields contained in the first data, and obtain a primary key value in the primary key field.

[0045] Exemplarily, the first data may be a piece of student information, which may include multiple fields such as a name field, a student ID field, and a class field. The student ID field may be the primary key field, and the primary key value contained in the primary key field is the student ID, such as 20190322, which can be used to uniquely identify each piece of student information.

[0046] Exemplarily, the first data may be an employee information, which may include multiple fields such as a name field, a position field, and a contact information field. The name field may be a primary key field. The primary key value contained in the primary key field is the employee's name, which can be used to uniquely identify each employee information.

[0047] Furthermore, if Figure 3 As shown, the data checking device 300 can reversely search the target end system 200 based on the primary key value of the first data to find the second data related to the primary key value from the data contained in the target end system 200. The second data also contains the primary key value, for example, the second data contains the same student number or name as the first data. Accordingly, the second data can be data synchronized from the source end system 100, specifically, the first data in the source end system 100 can be subjected to a series of conversion processes and finally written into the target end system 200.

[0048] Furthermore, if Figure 3 As shown, the data checking device 300 can check whether the first data and the second data are the same, and determine the data consistency between the source end system 100 and the target end system 200 based on the checking result, so as to perform corresponding data management on the target end system 200 later. For details, please refer to the following Figure 2 The description of the corresponding embodiments will not be elaborated in detail here.

[0049] It should be noted that the "first data" and "second data" described in this specification are only used to distinguish data from different systems, and do not specifically refer to a certain fixed data. That is to say, the data extracted from the source end system 100 can be collectively referred to as "first data", and the data extracted from the target end system 200 can be collectively referred to as "second data".

[0050] In one illustrated embodiment, Figure 1 The data checking device 300 shown may be a server having the above functions, or a server cluster composed of multiple servers, or may be a part of a device in a server, etc. This specification does not make any specific limitation on this.

[0051] It should be noted that Figure 1 The system architecture shown is for illustrative purposes only. In some possible implementations, the system architecture may also include other devices, for example, multiple other source end systems 100, or multiple other target end systems 200, etc. This specification does not make any specific limitations on this.

[0052] For further information, see Figure 2 , Figure 2is another system architecture diagram provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 2 The system architecture shown in the figure or a similar system architecture is specifically implemented. Figure 2 The internal structure of the data checking device 300 shown is used to explain the data checking method provided in this specification.

[0053] like Figure 2 As shown, the data verification device 300 may include a data extractor (Extractor), a data batch (Databatch) component, a data converter (Transformer), a data loader (Loader) and a data verifier (Verify).

[0054] like Figure 2 As shown, the data extractor is connected to the source end system and can extract data (or read data) from the source end system, such as the first data mentioned above.

[0055] In an illustrated embodiment, the data extractor may extract multiple first data in batches from the source end system according to a preset extraction rule. Furthermore, the data extractor may divide the extracted multiple first data into at least one batch and assign a sequence number to each batch. Each batch may contain at least part of the first data (one or more first data) among the multiple first data. In an illustrated embodiment, one or more first data may be organized into a batch through a selector, and this specification does not specifically limit this.

[0056] Furthermore, if Figure 2 As shown, the data extractor can write each batch of the at least one batch into the preconfigured Data batch component in order according to the respective sequence numbers of the at least one batch.

[0057] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a data batch component provided by an exemplary embodiment. Figure 3 As shown, the Data batch component is used to store batches organized by the Selector. A Data batch component can contain one or more batches, and each batch has a unique serial number (for example Figure 31, 2, ..., N shown). In an illustrated embodiment, each batch written into the Data batch component may undergo a subsequent data processing process in sequence, that is, subsequent data conversion, data loading, data verification, and other processes may be performed on each batch in the Data batch component in sequence according to the batch sequence number.

[0058] Furthermore, after the data extractor successfully writes the batch into the Data batch component, it can notify the next module in the data checking device 300 to perform the next step of processing. Figure 3 As shown, the data extractor can send a signal to the data converter to notify the data converter to perform the next step of processing, and the signal can include the serial number of the specified batch. In an embodiment shown, any other possible method other than the signal can also be used to send the notification, which is not specifically limited in this specification, and the following embodiments will be described by taking the signal as an example.

[0059] In an illustrated embodiment, the data converter may perform format conversion processing on the first data in the source end system to adapt to the format and requirements of the target end system. Specifically, Figure 2 As shown, after receiving the signal sent by the data extractor, the data converter can parse the signal to obtain the batch number contained in the signal. Then, the data converter can find the specified batch from the Data batch component according to the batch number, and perform format conversion processing on the first data contained in the batch according to the conversion rule configured by the user. In an embodiment shown, assuming that the data format of the data in the target end system is the target format, the data converter can perform format conversion processing on the first data in the batch based on the target format, thereby obtaining the first data in the target format.

[0060] Furthermore, after successfully converting the first data in the batch into the first data in the target format, the data converter may notify the next module in the data checking device 300 to perform the next step of processing. Figure 2 As shown, the data converter can send a signal to the data loader to notify the data loader to perform the next step of processing. Similarly, the signal can include the sequence number of the specified batch.

[0061] In an illustrated embodiment, the data loader may write the processing result of the data converter, that is, the first data in the target format, into the target end system. Figure 2As shown, after receiving the signal sent by the data converter, the data loader can parse the signal to obtain the batch number contained in the signal. Then, the data loader can find the specified batch from the Data batch component according to the batch number and write the first data in the target format contained in the batch into the target end system.

[0062] Furthermore, after the data loader successfully writes the first data in the target format in the batch into the target end system, it can notify the next module in the data checking device 300 to perform the next step of processing. Figure 2 As shown, the data loader may send a signal to the data verifier to notify the data verifier to perform the next step of processing. Similarly, the signal may include the sequence number of the specified batch.

[0063] In one embodiment, the data verifier can verify the consistency of data in the source end system and the target end system. Figure 2 As shown, after receiving the signal sent by the data loader, the data verifier can parse the signal to obtain the batch number contained in the signal. Then, the data verifier can find the specified batch from the Databatch component according to the batch number and obtain the primary key value in the batch.

[0064] In an illustrated embodiment, as described above, a batch may include one or more first data, each first data may include multiple fields, and these multiple fields may include a primary key field. Therefore, when the data verifier obtains the primary key value in the batch, it may first determine the primary key field of one or more first data included in the batch, and then obtain the primary key value in the primary key field of each first data.

[0065] Furthermore, if Figure 2 As shown, after obtaining the primary key value of each first data contained in the batch, the data verifier can reverse lookup the target end system according to the obtained primary key value to find the second data related to the primary key value of each first data from the data contained in the target end system.

[0066] Furthermore, after finding the second data related to the primary key value of each first data, the data verifier can verify the first data with the corresponding second data to check the data consistency between the target end system and the source end system. It should be understood that at this time, the first data in the batch is the first data after data conversion processing, that is, the first data in the target format, and the second data in the target end system is also in the target format. Therefore, when verifying the first data and the second data, the first data in the target format and the second data in the target format are specifically verified, thereby ensuring the reliability and accuracy of the data verification.

[0067] For example, assuming that a batch contains 4 first data, and these 4 first data are student information of 4 students, wherein the primary key value is the student ID, then the data verifier can specifically obtain the student IDs of these 4 students (such as No. 15, No. 20, No. 22, and No. 40) when obtaining the primary key value in the batch. Furthermore, the data verifier can reversely search the target end system according to the obtained 4 student IDs to find 4 second data that also contain these 4 student IDs from the target end system, and then the data verifier can check these 4 first data with the corresponding 4 second data one by one. For example, the first data with the student number 15 is checked against the second data with the same student number 15, the first data with the student number 20 is checked against the second data with the same student number 20, the first data with the student number 22 is checked against the second data with the same student number 22, and the first data with the student number 40 is checked against the second data with the same student number 40, so as to check whether all fields in each first data and the corresponding second data are one-to-one corresponding to each other.

[0068] In an embodiment shown, the data verifier can also verify the consistency of part or all of the data in the source end system and the target end system according to a preset sampling rate. The sampling rate can range from 0 to 100%, and the user can set a suitable sampling rate according to the actual situation or his own needs, such as 20%, 40% or 50%, etc., which is not specifically limited in this specification.

[0069] A sampling rate of 100% indicates that all data is checked, that is, data verification is performed on each first data in each batch in the Data batch component. A sampling rate of 0 may indicate that the current user has no data verification requirement, that is, data verification is not performed. A sampling rate greater than 0 and less than 100% may indicate that partial data is checked, that is, data verification is performed on part of the first data in each batch in the Data batch component, or data verification may be performed on each first data in part of the batches in the Databatch component, and so on. This specification does not make any specific limitation on this.

[0070] In an illustrated embodiment, the data verifier may select part of the first data from all the first data included in the batch according to a preset sampling rate, wherein the proportion of the part of the first data in all the first data may be equal to the sampling rate. Further, the data verifier may obtain the primary key value in the primary key field of the part of the first data. Further, the data verifier may reversely search the target end system according to the primary key value of the part of the first data, so as to find the second data related to the primary key value of the part of the first data from the target end system, and verify the part of the first data with the corresponding second data.

[0071] It should be noted that this specification does not specifically limit the specific implementation method of selecting part of the first data from all the first data included in the batch. In an illustrated embodiment, part of the first data may be randomly selected from all the first data included in the batch, or part of the first data at the front or back of the batch may be selected, etc. This specification does not specifically limit this.

[0072] For example, it is also assumed that the batch contains 4 first data, and these 4 first data are student information of 4 students, wherein the primary key value is the student number, and it is assumed that the sampling rate is 50%, then the data verifier can randomly select 2 first data from the 4 first data contained in the batch, and obtain the primary key values ​​of these 2 first data, that is, obtain 2 student numbers (such as 15 and 40), and then, the data verifier can reversely search the target end system according to the obtained 2 student numbers to find 2 second data that also contain these 2 student numbers from the target end system, and then the data verifier can check these 2 first data with the corresponding 2 second data one by one. For example, the first data with the student number of 15 is checked with the second data with the student number of 15, and the first data with the student number of 40 is checked with the second data with the student number of 40, to check whether all fields of each first data and the corresponding second data are the same one by one.

[0073] In an illustrated embodiment, the data verifier may also select some batches from all batches included in the Data batch component according to a preset sampling rate, wherein the proportion of the some batches in all batches may be equal to the sampling rate. Exemplarily, assuming the sampling rate is 50%, the data verifier may select 1 batch from every 2 batches of the Data batch component; assuming the sampling rate is 75%, the data verifier may select 3 batches from every 4 batches of the Data batch component, and so on, which is not specifically limited in this specification. Furthermore, the data verifier may execute the data verification process as described above for each selected batch, which will not be described in detail here.

[0074] In an embodiment shown, after the data verifier verifies the first data and the second data, if it is determined that the first data and the second data are all identical, it can be determined that the first data and the second data are successfully verified. Exemplarily, still taking the example that both the first data and the second data are student information, if the field values ​​contained in all fields such as the name field, the student number field, and the class field in the first data and the second data are all identical, it can be determined that the first data and the second data are successfully verified.

[0075] In an illustrated embodiment, after the data verifier verifies the first data and the second data, if it is determined that there is at least a partial difference between the first data and the second data, it can be determined that the verification of the first data and the second data has failed. Exemplarily, still taking the example that both the first data and the second data are student information, if the field values ​​contained in the student ID field and the class field in the first data and the second data are the same, but the field values ​​(i.e., the student names) contained in the name field of the first data and the second data are different, for example, the field value contained in the name field of the first data is "Zhang San", and the field value contained in the name field of the second data is "Li San", then it can be determined that the verification of the first data and the second data has failed.

[0076] In an illustrated embodiment, the present application takes into account that due to some possible technical failures or accidents, the data verifier may mistakenly determine that the originally identical first data and second data are failed to be verified, thereby causing an error in the data verification result and affecting subsequent data management. Therefore, the present application further designs a fault-tolerant mechanism for data verification, and the data verifier can repeatedly verify the first data and the second data that failed to be verified within a preset tolerance window. If it is still determined that there is at least a partial difference between the first data and the second data after reaching the preset tolerance window, it can be finally determined that the first data and the second data have failed to be verified.

[0077] In an illustrated embodiment, the preset tolerance window may be a preset verification time or a preset number of verifications, which is not specifically limited in this specification. Specifically, after the data verifier determines that there is at least a partial difference between the first data and the second data, the first data and the second data may be verified again. Further, if it is still determined that there is at least a partial difference between the first data and the second data after the re-verification, the first data and the second data may continue to be verified, and so on, until the preset verification time or the preset number of verifications is reached, and the verification is terminated. Further, if it is still determined that there is at least a partial difference between the first data and the second data after the preset verification time is reached or after the preset number of verifications is reached, it can be finally determined that the verification of the first data and the second data has failed; otherwise, if it is determined that the first data and the second data are completely identical within the preset verification time or within the preset number of verifications, it can be finally determined that the verification of the first data and the second data has been successful.

[0078] It should be understood that if the data verifier cannot find the second data related to the primary key value of the first data from the target end system (for example, it cannot find the second data with the student number 15), it can also be determined that the first data verification has failed. In an illustrated embodiment, if the data verifier cannot find the second data related to the primary key value of the first data from the target end system within a preset search time or search times, it can be determined that the first data verification has failed.

[0079] In an illustrated embodiment, the data verifier may also calculate the verification failure rate according to the number of verified first data and the number of first data that failed to be verified. Specifically, the proportion of the number of first data that failed to be verified in the number of all verified first data may be calculated, and the proportion may be determined as the verification failure rate.

[0080] Furthermore, the data verifier can determine whether the calculated verification failure rate is greater than a preset threshold. If the verification failure rate is greater than the preset threshold, the current data integration can be stopped, that is, the synchronization of the data of the source end system to the target end system can be stopped, so as to stop the loss in time and avoid expanding the data quality problem of the target end system. In an illustrated embodiment, if the verification failure rate is greater than the preset threshold, the data verifier can send a signal to the data loader to notify the data loader to stop writing the first data in the target format contained in the batch into the target end system.

[0081] In an illustrated embodiment, the user can set the preset threshold of the above-mentioned verification failure rate according to actual conditions and needs, for example, it can be 10%, 15% or 35%, etc., and this specification does not make specific limitations on this. For example, if the preset threshold of the verification failure rate is set to 0, it can mean that the verification failure rate is ignored and does not affect data integration; or, the preset threshold can be set to 100% to indicate that the verification failure rate is ignored, and this specification does not make specific limitations on this.

[0082] In one illustrated embodiment, after determining that the first data and the second data have failed to be verified, the data verifier may also store the first data that failed to be verified, the second data and the verification results thereof in a preset database, or in other systems, so that the user can analyze and find data anomalies, and perform corresponding data management operations on the target end system to ensure the data integrity and accuracy of the target end system, etc. This specification does not make any specific limitations on this.

[0083] In an illustrated implementation, considering that data integration is concurrent writing, for example, multiple data update operations in the source system will be concurrently written into the target system, and the frequently modified data will lead to inconsistent verification in a short period of time, thereby increasing the verification failure rate. For example, assuming that the first data extracted from the source system is the account information of account A, if account A transfers money to account B multiple times in a short period of time, the balance of account A will change frequently, which will easily lead to the balance field in the second data currently found being different from that in the first data.

[0084] Based on this, the present application also provides a data review method, which can periodically review the failed verification data stored in the above-mentioned preset database to further reduce the verification failure rate.

[0085] In one illustrated embodiment, see Figure 4 , Figure 4 FIG. 1 is a flow chart of a data review method provided by an exemplary embodiment. Figure 4 As shown, the data verifier can periodically obtain the primary key value of the data that failed to be verified from the preset database based on the preset periodic duration. Further, the data verifier can reversely search the source end system and the target end system based on the obtained primary key value to find the source data related to the primary key value from the source end system, and find the target data related to the primary key value from the target end system. Further, the data verifier can verify the source data and the target data, and perform corresponding data management on the target end system based on the verification result.

[0086] In an illustrated embodiment, after the data verifier verifies the source data and the target data, if it is determined that the source data and the target data are at least partially different, it can be determined that there is indeed an abnormality in the target data, and the target data in the target end system can be modified to make the modified target data the same as the source data, thereby ensuring the data integrity and accuracy of the target end system, thereby providing users with reliable data services.

[0087] In an illustrated embodiment, after the data verifier verifies the source data and the target data, if it is determined that the source data and the target data are all the same, the data that originally failed to be verified and its verification results can be deleted from the preset database, or the verification results can be modified in the preset database to re-record the data that originally failed to be verified as the data that successfully verified, etc., and this specification does not make specific limitations on this. Exemplarily, after a period of time, the above-mentioned account A no longer transfers money to account B, and all previous balance changes of account A have been successfully written into the target end system. At this time, the source end data and the target end data corresponding to account A are all the same, then the first data and the second data corresponding to account A that originally failed to be verified and their verification results can be deleted from the preset database, or its verification results can be modified to be verified successfully, etc., and this specification does not make specific limitations on this.

[0088] In summary, this application adds the ability of data verification on the basis of ETL and realizes ETLV ((Extract, Transform, Load, Verify). This application not only focuses on the data integration process, but also emphasizes the data consistency between the source system and the target system. By performing data verification during or after the data integration process, the data consistency between the target system and the source system can be verified. This process not only involves data quality inspection, but also includes data auditing and verification to ensure the consistency of data in content, structure and format. Data consistency verification not only improves data quality, but also ensures data consistency and compliance, thereby increasing the credibility of data, helping users to use data with confidence and ensuring user experience.

[0089] Once again, data consistency verification can not only confirm the integrity of data during synchronization or migration, but also help identify potential data quality issues, thereby improving the overall data governance level and providing a more reliable basis for decision-making for enterprises. Ultimately, through efficient data integration and strict data verification mechanisms, more efficient data processing, more reliable data analysis and more secure data management can be provided, providing strong support for the data-driven decision-making of enterprises, enabling enterprises to use data analysis to gain greater strategic advantages in the fierce market competition.

[0090] In one illustrated embodiment, Figure 2The data extractor, data batch component, data converter, data loader and data verifier shown may be implemented by software, hardware or a combination of software and hardware, and this specification does not impose any specific limitation on this.

[0091] In addition, it should be noted that Figure 2 The structure of the data checking device 300 shown is only for illustrative purposes. In some possible implementations, the data checking device 300 may include a Figure 2 There are more or fewer components shown, and this specification does not make any specific limitations on this.

[0092] For example, if the user does not need to perform real-time data verification during the data integration process, or the user only needs to verify the data between the source end system and the target end system after data integration, then the Figure 2 The data loader in the data integration device, accordingly, can independently implement the data integration function by other data integration devices, and the data verification device 300 provided in the present application can only implement the data verification function. In an illustrated embodiment, the data verification device 300 can regularly perform data verification on the source end system and the target end system at certain time intervals to reduce system pressure. Exemplarily, the data extractor can periodically extract a certain amount of first data from the source end database according to a preset cycle duration, organize one or more first data into a batch, and write the batch into the Data batch component. Further, the data extractor can send a signal to the data converter so that the data converter converts the first data in the batch into the first data in the target format supported by the target end system. Further, the data converter can directly send a signal to the data verifier so that the data verifier executes the data verification process as described above for the first data in the target format contained in the batch. For details, please refer to the above Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0093] For example, if the data format of the data in the target system is the same as that of the data in the source system, and the data structure is relatively simple, then Figure 2 Correspondingly, after writing the batch into the Data batch component, the data extractor can directly send a signal to the data loader to enable the data loader to write the first data contained in the batch into the target end system, and so on. The subsequent process can refer to the above Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0094] For example, if the user only needs to perform data verification periodically, and the data format of the data in the target system is the same as that of the data in the source system, then the Figure 2 Correspondingly, after writing the batch into the Data batch component, the data extractor can directly send information to the data verifier so that the data verifier performs the above-mentioned data verification process on the first data included in the batch. For details, please refer to the above Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0095] For example, if the user only needs to perform data integration and does not need to perform data verification, this can be omitted. Figure 2 Correspondingly, after the data loader writes the first data in the target format contained in the batch into the target end system, the process can be ended.

[0096] In summary, users can choose to use ETLV, ETV, ELV, EV or ETL according to actual conditions and needs, and this manual does not make specific limitations on this.

[0097] See also Figure 5 , Figure 5 is a flow chart of a data verification method provided by an exemplary embodiment. The method can be applied to Figure 1 or Figure 2 The system architecture shown can be applied to Figure 1 or Figure 2 The data checking device 300 in the system architecture shown. Figure 5 As shown, the method may specifically include the following steps S501 to S504.

[0098] Step S501: extract first data from the data contained in the source end system, where the first data contains a plurality of fields.

[0099] Step S502: determine a primary key field among the multiple fields included in the first data, and obtain a primary key value in the primary key field.

[0100] Step S503, searching for second data related to the primary key value from the data contained in the target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data includes the second data.

[0101] Step S504: check the first data and the second data, and perform corresponding data management on the target end system based on the check result.

[0102] The specific implementation of the above steps S501 to S504 can refer to the above Figure 1-Figure 4 The description of the corresponding embodiments will not be repeated here.

[0103] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a data verification device. Figure 6 , Figure 6 is a schematic diagram of a data checking device provided by an exemplary embodiment. The device 60 can be applied to Figure 1 or Figure 2 The system architecture shown can be applied to Figure 1 or Figure 2 The data checking device 300 in the system architecture shown. Figure 6 As shown, the device 60 includes:

[0104] The source end data extraction unit 601 is used to extract first data from the data contained in the source end system, where the first data contains a plurality of fields;

[0105] A primary key value acquiring unit 602 is used to determine a primary key field among the multiple fields included in the first data, and acquire a primary key value in the primary key field;

[0106] The target-side data extraction unit 603 is configured to search for second data related to the primary key value from the data contained in the target-side system; wherein at least part of the data in the target-side system is data synchronized from the source-side system, and the at least part of the data includes the second data;

[0107] The data checking unit 604 is used to check the first data and the second data, and perform corresponding data management on the target end system based on the checking result.

[0108] In an illustrated embodiment, the source data extraction unit 601 is specifically used to:

[0109] Extracting a plurality of first data in batches from the source end system;

[0110] Divide the plurality of first data into at least one batch, and assign a sequence number to each batch; wherein each batch includes at least part of the plurality of first data;

[0111] Each batch in the at least one batch is sequentially written into the preconfigured data component in order according to the respective sequence numbers of the at least one batch.

[0112] In an illustrated embodiment, the primary key value obtaining unit 602 is specifically configured to:

[0113] In the preconfigured data component, the primary key field of the first data contained in each batch is determined in sequence according to the sequence number of the batch, and the primary key value in the primary key field of the first data is obtained.

[0114] In an illustrated embodiment, the primary key value obtaining unit 602 is specifically configured to:

[0115] According to a preset sampling rate, randomly select part of the first data from all the first data included in the batch; wherein the proportion of the part of the first data in all the first data is equal to the sampling rate;

[0116] A primary key value in a primary key field of the portion of first data is obtained.

[0117] In an illustrated embodiment, the data checking unit 604 is specifically used for:

[0118] Verifying the first data and the second data, and if the first data and the second data are at least partially different, determining that the verification of the first data and the second data fails;

[0119] storing the first data, the second data and the verification result thereof in a preset database;

[0120] Based on the data that failed to be verified and stored in the preset database, corresponding data management is performed on the target end system.

[0121] In an illustrated embodiment, the data checking unit 604 is specifically used for:

[0122] If the first data and the second data are at least partially different, then the first data and the second data are checked again. If it is still determined after the re-check that the first data and the second data are at least partially different, then the first data and the second data are checked again, and so on, until a preset checking time is reached or a preset number of checking times is reached;

[0123] If it is still determined that there is at least a partial difference between the first data and the second data after reaching a preset verification time or after reaching a preset number of verifications, it is further determined that the verification of the first data and the second data has failed.

[0124] In an illustrated embodiment, the data checking unit 604 is specifically used for:

[0125] Based on a preset periodic duration, periodically obtaining the primary key value of the data that failed to be verified from the preset database;

[0126] Searching for source data related to the primary key value from the source end system, and searching for target data related to the primary key value from the target end system;

[0127] The source data and the target data are checked, and corresponding data management is performed on the target end system based on the check result.

[0128] In an illustrated embodiment, the data checking unit 604 is specifically used for:

[0129] The source data and the target data are checked, and if the source data and the target data are at least partially different, the target data in the target end system is modified so that the modified target data is the same as the source data.

[0130] In an illustrated embodiment, the device 60 further includes a data modification unit 605, which is used to:

[0131] If the source data and the target data are all the same, the data that failed to be verified are deleted from the preset database, or the data that failed to be verified are re-recorded as data that succeeded to be verified in the preset database.

[0132] In an illustrated embodiment, the data format of the data in the source end system is different from the data format of the data in the target end system; the data checking unit 604 is specifically used to:

[0133] Based on the target format of the data in the target end system, performing format conversion processing on the first data to obtain the first data in the target format; wherein the data format of the second data is the target format;

[0134] The first data in the target format is collated with the second data in the target format.

[0135] In an illustrated embodiment, the device 60 further includes a data loading unit 606, which is used to:

[0136] The first data in the target format is written into the target end system.

[0137] In an illustrated embodiment, the device 60 further includes a verification failure rate calculation unit 607, which is used to:

[0138] Calculating a verification failure rate according to the number of verified first data and the number of first data that failed verification;

[0139] Determine whether the verification failure rate is greater than a preset threshold, and if the verification failure rate is greater than the preset threshold, stop data synchronization between the source end system and the target end system.

[0140] The implementation process of the functions and effects of each unit in the above-mentioned device 60 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 60 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0142] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0143] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 7 , Figure 7 It is a structural diagram of a computing device provided by an exemplary embodiment. Figure 7 The computing device shown may be Figure 1 or Figure 2 The data checking device 300 in the system architecture shown. Figure 7 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 7As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the data verification method in the embodiment of this specification, including: extracting first data from the data contained in the source end system, the first data contains multiple fields; determining the primary key field in the multiple fields contained in the first data, and obtaining the primary key value in the primary key field; finding the second data related to the primary key value from the data contained in the target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data contains the second data; verifying the above-mentioned first data and second data, and performing corresponding data management on the target end system based on the verification result, etc.

[0144] For a detailed description of each step of the above data verification method, please refer to the previous content and will not be repeated here.

[0145] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when these computer programs are executed by a processor, the various steps of the data verification method in the embodiment of this specification are executed. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0146] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0147] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0148] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0149] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0150] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0152] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A data verification method, characterized in that: The method comprises: Extracting first data from data contained in a source end system, wherein the first data includes a plurality of fields; Determine a primary key field among the multiple fields included in the first data, and obtain a primary key value in the primary key field; Finding second data related to the primary key value from the data contained in the target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data includes the second data; The first data and the second data are checked, and corresponding data management is performed on the target end system based on the checking result.

2. The method according to claim 1, characterized in that The step of extracting the first data from the data contained in the source end system includes: Extracting a plurality of first data in batches from the source end system; Divide the plurality of first data into at least one batch, and assign a sequence number to each batch; wherein each batch includes at least part of the plurality of first data; Each batch in the at least one batch is sequentially written into the preconfigured data component in order according to the respective sequence numbers of the at least one batch.

3. The method according to claim 2, characterized in that The determining of a primary key field among the multiple fields included in the first data and obtaining a primary key value in the primary key field includes: In the preconfigured data component, the primary key field of the first data contained in each batch is determined in sequence according to the sequence number of the batch, and the primary key value in the primary key field of the first data is obtained.

4. The method according to claim 3, characterized in that The obtaining the primary key value in the primary key field of the first data includes: According to a preset sampling rate, randomly select part of the first data from all the first data included in the batch; wherein the proportion of the part of the first data in all the first data is equal to the sampling rate; A primary key value in a primary key field of the portion of first data is obtained.

5. The method according to claim 3, characterized in that: The checking of the first data and the second data, and performing corresponding data management on the target end system based on the checking result, includes: Verifying the first data and the second data, and if the first data and the second data are at least partially different, determining that the verification of the first data and the second data fails; storing the first data, the second data and the verification result thereof in a preset database; Based on the data that failed to be verified and stored in the preset database, corresponding data management is performed on the target end system.

6. The method according to claim 5, characterized in that If the first data and the second data are at least partially different, determining that the first data and the second data have failed to be verified includes: If the first data and the second data are at least partially different, then the first data and the second data are checked again. If it is still determined after the re-check that the first data and the second data are at least partially different, then the first data and the second data are checked again, and so on, until a preset checking time is reached or a preset number of checking times is reached; If it is still determined that there is at least a partial difference between the first data and the second data after reaching a preset verification time or after reaching a preset number of verifications, it is further determined that the verification of the first data and the second data has failed.

7. The method according to claim 5, characterized in that The performing corresponding data management on the target end system based on the data that failed to be verified and stored in the preset database includes: Based on a preset periodic duration, periodically obtaining the primary key value of the data that failed to be verified from the preset database; Searching for source data related to the primary key value from the source end system, and searching for target data related to the primary key value from the target end system; The source data and the target data are checked, and corresponding data management is performed on the target end system based on the check result.

8. The method according to claim 7, characterized in that The checking of the source data and the target data, and performing corresponding data management on the target end system based on the checking result, includes: The source data and the target data are checked, and if the source data and the target data are at least partially different, the target data in the target end system is modified so that the modified target data is the same as the source data.

9. The method according to claim 8, characterized in that The method further comprises: If the source data and the target data are all the same, the data that failed to be verified are deleted from the preset database, or the data that failed to be verified are re-recorded as data that succeeded to be verified in the preset database.

10. The method according to claim 5, characterized in that The data format of the data in the source end system is different from the data format of the data in the target end system; The checking of the first data and the second data includes: Based on the target format of the data in the target end system, performing format conversion processing on the first data to obtain the first data in the target format; wherein the data format of the second data is the target format; The first data in the target format is collated with the second data in the target format.

11. The method according to claim 10, characterized in that Before searching for second data related to the primary key value from the data included in the target end system, the method further includes: The first data in the target format is written into the target end system.

12. The method according to claim 11, characterized in that The method further comprises: Calculating a verification failure rate according to the number of verified first data and the number of first data that failed verification; Determine whether the verification failure rate is greater than a preset threshold, and if the verification failure rate is greater than the preset threshold, stop data synchronization between the source end system and the target end system.

13. A data checking device, characterized in that: The device comprises: A source-end data extraction unit, configured to extract first data from data contained in a source-end system, wherein the first data includes a plurality of fields; a primary key value acquiring unit, configured to determine a primary key field among the multiple fields included in the first data, and acquire a primary key value in the primary key field; A target-side data extraction unit, configured to search for second data related to the primary key value from data contained in the target-side system; wherein at least part of the data in the target-side system is data synchronized from the source-side system, and the at least part of the data includes the second data; A data checking unit is used to check the first data and the second data, and perform corresponding data management on the target end system based on the checking result.

14. A computing device, characterized in that include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, it performs the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images