A data reconciliation method and related device

Abstract

The specification provides a data checking method and related equipment. The method comprises: extracting first data from data contained in a source end system, the first data containing a plurality of fields; determining a primary key field in the plurality of fields contained in the first data, and obtaining a primary key value in the primary key field; finding second data related to the primary key value from data contained in a target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data contains the second data; checking the first data and the second data, and performing corresponding data management on the target end system based on a checking result.

Description

Technical Field

[0001] This specification relates to the field of data management technology, and in particular to a data verification method and related equipment. Background Technology

[0002] ETL (Extract, Transform, Load) is a data integration technology primarily used to extract data from source systems, transform the extracted data to suit the format or requirements of target systems, and finally load the processed data into the target systems. ETL technology plays a crucial role in modern data management, helping enterprises and organizations integrate and process data from diverse sources, and providing a solid foundation for advanced analytics and decision support. Whether building data warehouses, managing data lakes, or achieving data synchronization and migration, ETL is an indispensable technical tool.

[0003] However, in traditional ETL technology, complex data transformation rules and logic can lead to inconsistencies in data changes, thus affecting the accuracy of the final data integration results. For example, during data transformation processing (which may include a series of complex processes such as data normalization or merging), technical defects or human errors may cause the loss of some or even all of the data, resulting in incomplete or inaccurate data in the target system, making it unable to maintain consistency with the data in the source system. These data quality issues not only affect corporate decision-making but may also bring serious compliance risks.

[0004] Therefore, ensuring data consistency between the target system and the source system based on ETL technology is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a data verification method and related equipment.

[0006] Firstly, this specification provides a data verification method, the method comprising:

[0007] Extract the first data from the data contained in the source system. The first data contains multiple fields.

[0008] Determine the primary key field among the plurality of fields contained in the first data, and obtain the primary key value in the primary key field;

[0009] Find the second data related to the primary key value from the data contained in the target system; wherein at least a portion of the data in the target system is data synchronized from the source system, and the at least a portion of the data contains the second data;

[0010] The first data is compared with the second data, and the corresponding data management is performed on the target system based on the comparison result.

[0011] Secondly, this specification provides a data verification device, the device comprising:

[0012] The source data extraction unit is used to extract first data from the data contained in the source system, the first data containing multiple fields;

[0013] A primary key value acquisition unit is used to determine the primary key field among the plurality of fields contained in the first data, and to acquire the primary key value in the primary key field;

[0014] A target-end data extraction unit is used to find second data related to the primary key value from the data contained in the target-end system; wherein at least a portion of the data in the target-end system is data synchronized from the source-end system, and the at least a portion of the data includes the second data;

[0015] The data verification unit is used to verify the first data and the second data, and to perform corresponding data management on the target terminal system based on the verification result.

[0016] Accordingly, this specification also provides a computing device, including: a memory and a processor; the memory stores a computer program / instructions executable by the processor; when the processor executes the computer program / instructions, it performs the data verification method described in the first aspect above.

[0017] Accordingly, this specification also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, performs the data verification method as described in the first aspect above.

[0018] Accordingly, this specification also provides a computer program product comprising a computer program / instruction, which, when executed by a processor, performs the data verification method as described in the first aspect above.

[0019] In summary, after extracting the first data from the source system, this application can determine the primary key field among the multiple fields contained in the first data, obtain the primary key value of the primary key field, and then perform a reverse lookup on the target system based on the primary key value to find the second data related to the primary key value in the target system. Finally, the first data and the second data are verified to confirm the data consistency between the target system and the source system. Thus, by adding an additional data verification step, this application can verify the data consistency between the target system and the source system based on data integration performed from the source system to the target system (e.g., synchronizing data from the source system to the target system), thereby ensuring the data integrity and accuracy of the target system and providing users with more reliable and secure data services. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a system architecture provided in an exemplary embodiment;

[0021] Figure 2 This is a schematic diagram of another system architecture provided in an exemplary embodiment;

[0022] Figure 3 This is a schematic diagram of the structure of a data batching component provided in an exemplary embodiment;

[0023] Figure 4 This is a flowchart illustrating a data verification method provided in an exemplary embodiment;

[0024] Figure 5 This is a flowchart illustrating a data verification method provided in an exemplary embodiment;

[0025] Figure 6 This is a schematic diagram of the structure of a data verification device provided in an exemplary embodiment;

[0026] Figure 7 This is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0029] It should be noted that "multiple" in this application refers to two or more.

[0030] Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0031] ETL is a data integration technology widely used in various scenarios such as data warehousing and big data processing workflows. It includes three core steps:

[0032] Extract: Extract raw data from one or more source systems. The source system can be any possible data source, such as relational databases, non-relational databases, CSV files, XML files, and API interfaces, etc. This specification does not specify any particular source.

[0033] Transform: This step involves cleaning, transforming, and normalizing the raw data extracted from the source system to adapt it to the format and requirements of the target system. This step may include operations such as filling in default values, extracting data that meets the criteria, removing duplicate records, data type conversion, data aggregation, and data quality checks, which are not specifically limited in this specification.

[0034] Load: Loads the transformed data (i.e., the result of the Transform) into the target system, which can also be a data warehouse, data lake, or any other possible system, to facilitate subsequent analysis and report generation. The target system can be a relational database, a non-relational database, a CSV file, an XML file, or an API interface, etc., and this specification does not specify any particular type.

[0035] As service demands increase and data complexity grows, ETL will involve more complex data transformation rules and logic. These complex rules and logic can lead to inconsistencies in data changes, thus affecting the accuracy of the final data integration results. For example, during the transformation process, technical defects or human errors may result in the loss of some or even all of the data, leading to incomplete or inaccurate data in the target system, making it unable to maintain consistency with the source system. These data quality issues not only affect enterprise decision-making but may also bring serious compliance risks.

[0036] Based on this, this specification provides a technical solution that, through additional data verification steps, verifies the data consistency between the target system and the source system on the basis of data integration, thereby providing users with more reliable and secure data services.

[0037] In implementation, firstly, first data can be extracted from the data contained in the source system. This first data may contain multiple fields. Then, the primary key field among the multiple fields contained in the first data is determined, and its primary key value is obtained. Further, second data related to this primary key value is found from the data contained in the target system; wherein at least a portion of the data in the target system is synchronized from the source system, and this at least a portion of the data contains the second data. Finally, the first and second data can be verified, and corresponding data management can be performed on the target system based on the verification result.

[0038] In the above technical solution, after extracting the first data from the source system, this application can determine the primary key field among the multiple fields contained in the first data, obtain the primary key value in the primary key field, and then perform a reverse lookup on the target system based on the primary key value to find the second data related to the primary key value in the target system. Finally, the first data and the second data are verified to confirm the data consistency between the target system and the source system. Thus, by adding an additional data verification step, this application can verify the data consistency between the target system and the source system based on data integration performed from the source system to the target system (e.g., synchronizing data from the source system to the target system), thereby ensuring the data integrity and accuracy of the target system and providing users with more reliable and secure data services.

[0039] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided in an exemplary embodiment. One or more embodiments provided in this specification can be used... Figure 1 The specific implementation is within the system architecture shown or a similar system architecture. For example... Figure 1As shown, the system architecture may include a source system 100, a target system 200, and a data verification device 300. The source system 100, the target system 200, and the data verification device 300 can establish communication connections in any possible manner; this specification does not impose specific limitations on this.

[0040] In this context, the source system 100 and the target system 200 can be two ends of the data integration process. That is, at least some of the data in the target system 200 is synchronized from the source system 100.

[0041] It should be noted that this manual does not specifically limit the specific scenarios for data integration.

[0042] In one illustrated implementation, the data integration scenario can be a primary / backup database synchronization scenario. The source system 100 can be the primary database in a primary / backup database system, and the target system 200 can be the backup database in the primary / backup database system. Data in the primary database can be synchronized to the backup database in real time or periodically to achieve data backup. This ensures that in the event of an unrecoverable failure in the primary database, services can be quickly switched to the backup database, thereby continuing to provide services to users and ensuring that the user experience is not affected.

[0043] In one illustrated implementation, the data integration scenario can be a data warehouse or data lake synchronization scenario. The source system 100 can be various data sources (e.g., sales systems or inventory systems of various enterprises), and the target system 200 can be a data warehouse or data lake. Data is extracted from various data sources and synchronized to the data warehouse or data lake for analysis and report generation, etc. This specification does not specifically limit this.

[0044] like Figure 3 As shown, in one illustrated embodiment, the data verification device 300 can first extract first data from the data contained in the source system 100, the first data potentially containing multiple fields. Further, the data verification device 300 can determine the primary key field among the multiple fields contained in the first data and obtain the primary key value from that primary key field.

[0045] For example, the first piece of data can be a student information record, which may contain multiple fields such as name, student ID, and class. The student ID field can be a primary key field, and the primary key value contained in the primary key field is the student ID, such as 20190322, which can be used to uniquely identify each student information record.

[0046] For example, the first piece of data can be a piece of employee information, which may include multiple fields such as name field, job title field and contact information field. The name field can be a primary key field, and the primary key value contained in the primary key field is the employee's name, which can be used to uniquely identify each piece of employee information.

[0047] Furthermore, such as Figure 3 As shown, the data verification device 300 can reverse-search the target system 200 based on the primary key value of the first data to find the second data related to the primary key value from the data contained in the target system 200. The second data also contains the primary key value; for example, the second data may contain a student ID or name identical to the first data. Correspondingly, the second data can be data synchronized from the source system 100, specifically, it can be data written to the target system 200 after a series of transformations of the first data in the source system 100.

[0048] Furthermore, such as Figure 3 As shown, the data verification device 300 can verify whether the first data and the second data are the same, and determine the data consistency between the source system 100 and the target system 200 based on the verification result, so as to perform corresponding data management on the target system 200 in the future. For details, please refer to the following. Figure 2 The description of the corresponding embodiments will not be elaborated here.

[0049] It should be noted that the terms "first data" and "second data" used in this specification are only used to distinguish data from different systems and do not refer to any specific fixed data. In other words, data extracted from the source system 100 can be collectively referred to as "first data" and data extracted from the target system 200 can be collectively referred to as "second data".

[0050] In one illustrated embodiment, Figure 1 The data verification device 300 shown may be a server with the above-mentioned functions, or a server cluster consisting of multiple servers, or it may be a part of a server, etc. This specification does not make any specific limitations on it.

[0051] It should be noted that, Figure 1 The system architecture shown is for illustrative purposes only. In some possible implementations, the system architecture may also include other devices, such as multiple other source end systems 100 or multiple other target end systems 200, etc. This specification does not specifically limit this.

[0052] Further, please refer to Figure 2 , Figure 2This is a schematic diagram of another system architecture provided in an exemplary embodiment. One or more embodiments provided in this specification can be used... Figure 2 The specific implementation will be carried out in the system architecture shown or a similar system architecture. Below, we will combine... Figure 2 The internal structure of the data verification device 300 shown is illustrated, and the data verification method provided in this specification is explained.

[0053] like Figure 2 As shown, the data verification device 300 may include a data extractor, a data batch component, a data transformer, a data loader, and a data verifier.

[0054] like Figure 2 As shown, the data extractor interfaces with the source system and can extract (or read) data from the source system, such as the first data mentioned above.

[0055] In one illustrated embodiment, the data extractor can extract multiple first data sets from the source system in batches according to preset extraction rules. Further, the data extractor can divide the extracted multiple first data sets into at least one batch and assign a sequence number to each batch. Each batch may contain at least a portion (one or more) of the multiple first data sets. In one illustrated embodiment, a selector can be used to organize one or more first data sets into a batch; this specification does not specifically limit this approach.

[0056] Furthermore, such as Figure 2 As shown, the data extractor can sequentially write each batch in the at least one batch into the pre-configured Data batch component according to the sequence number of each batch.

[0057] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a data batching component provided in an exemplary embodiment. For example... Figure 3 As shown, the Data batch component is used to store batches organized by the Selector. A Data batch component can contain one or more batches, and each batch has a unique sequence number (e.g., ...). Figure 3(as shown in 1, 2, ..., N). In one illustrated implementation, each batch written into the Data batch component can sequentially undergo subsequent data processing flows, that is, according to the batch number, subsequent data transformation, data loading, and data verification processes can be performed on each batch in the Data batch component in sequence.

[0058] Furthermore, after successfully writing the batch to the Data batch component, the data extractor can notify the next module in the data verification device 300 to perform the next step of processing. For example... Figure 3 As shown, the data extractor can send a signal to the data converter to notify the data converter to perform the next processing step. This signal may contain the sequence number of the specified batch. In one illustrated embodiment, any possible method other than a signal can be used to send the notification; this specification does not specifically limit this method, and the following embodiments will all use signals as examples.

[0059] In one illustrated embodiment, the data converter can perform format conversion processing on first data in the source system to adapt it to the format and requirements of the target system. Specifically, such as... Figure 2 As shown, after receiving a signal from the data extractor, the data converter can parse the signal to obtain the batch number contained within it. Then, based on this batch number, the data converter can locate the specified batch from the Data batch component and perform format conversion processing on the first data contained in that batch according to the user-configured conversion rules. In one illustrated embodiment, assuming the data format in the target system is the target format, the data converter can perform format conversion processing on the first data in the batch based on the target format, thereby obtaining the first data in the target format.

[0060] Furthermore, after successfully converting the first data in the batch into the first data in the target format, the data converter can notify the next module in the data verification device 300 to perform the next step of processing. For example... Figure 2 As shown, the data converter can send a signal to the data loader to notify the data loader to perform the next step of processing. Similarly, the signal can contain the sequence number of the specified batch.

[0061] In one illustrated implementation, the data loader can write the processing result of the data converter, i.e., first data in the target format, into the target end system. Specifically, as... Figure 2As shown, after receiving the signal sent by the data converter, the data loader can parse the signal to obtain the batch number contained in the signal. Then, the data loader can find the specified batch from the Data batch component according to the batch number and write the first data in the target format contained in the batch into the target system.

[0062] Furthermore, after successfully writing the first data in the target format from the batch to the target system, the data loader can notify the next module in the data verification device 300 to perform the next step of processing. For example... Figure 2 As shown, the data loader can send a signal to the data checker to notify the data checker to perform the next step of processing. Similarly, the signal can contain the sequence number of the specified batch.

[0063] In one illustrated implementation, the data checker can verify the consistency of data between the source system and the target system. Specifically, as shown... Figure 2 As shown, after receiving the signal sent by the data loader, the data checker can parse the signal to obtain the batch number contained in the signal. Then, the data checker can find the specified batch from the Databatch component based on the batch number and obtain the primary key value in the batch.

[0064] In one illustrated implementation, as described above, a batch may contain one or more first data, and each first data may contain multiple fields, including a primary key field. Therefore, when the data checker obtains the primary key value in the batch, it may first determine the primary key fields of one or more first data contained in the batch, and then obtain the primary key value in the primary key field of each first data.

[0065] Furthermore, such as Figure 2 As shown, after obtaining the primary key value of each first data in the batch, the data checker can reverse lookup the target system based on the obtained primary key value to find the second data related to the primary key value of each first data in the data contained in the target system.

[0066] Furthermore, after locating the second data associated with the primary key value of each piece of first data, the data checker can compare the first data with the corresponding second data to verify the data consistency between the target system and the source system. It should be understood that the first data in the batch at this point is the first data after data transformation, i.e., the first data in the target format. The second data in the target system is also in the target format. Therefore, when verifying the first and second data, the system specifically verifies the first data in the target format and the second data in the target format, thus ensuring the reliability and accuracy of the data verification.

[0067] For example, suppose a batch contains four pieces of primary data, representing student information for four students, with the student ID as the primary key. The data checker, when retrieving the primary key from this batch, could specifically retrieve the student IDs of these four students (e.g., IDs 15, 20, 22, and 40). Further, the data checker can use these four student IDs to perform a reverse lookup in the target system to find four pieces of secondary data containing the same four student IDs. The data checker can then verify these four pieces of primary data against the corresponding four pieces of secondary data. For example, the first data for student ID 15 is compared with the second data for student ID 15, the first data for student ID 20 is compared with the second data for student ID 20, the first data for student ID 22 is compared with the second data for student ID 22, and the first data for student ID 40 is compared with the second data for student ID 40, in order to check whether all fields of each first data and the corresponding second data are identical.

[0068] In one illustrated embodiment, the data checker can also verify the consistency of some or all data in the source system and the target system according to a preset sampling rate. The sampling rate can range from 0% to 100%, and users can set a suitable sampling rate based on actual conditions or their own needs, such as 20%, 40%, or 50%, etc. This specification does not impose a specific limitation on this.

[0069] A sampling rate of 100% indicates that all data is checked, meaning that data verification is performed on every first data point in each batch of the Data batch component. A sampling rate of 0 indicates that the current user has no data verification requirement, meaning that no data verification is performed. A sampling rate greater than 0 and less than 100% can indicate that partial data is checked, meaning that data verification is performed on a portion of the first data points in each batch of the Data batch component, or it can be performed on every first data point in a portion of the batches of the Data batch component, etc. This specification does not specify any particular limitation.

[0070] In one illustrated embodiment, the data checker can select a portion of first data from all first data included in a batch according to a preset sampling rate, wherein the proportion of this portion of first data in all first data can be equal to the sampling rate. Further, the data checker can obtain the primary key value from the primary key field of this portion of first data. Further, the data checker can use the primary key value of this portion of first data to perform a reverse lookup in the target system to find second data related to the primary key value of this portion of first data in the target system, and verify the first data portion against the corresponding second data.

[0071] It should be noted that this specification does not specifically limit the method of selecting a portion of the first data from all the first data contained in the batch. In one illustrated implementation, a portion of the first data may be randomly selected from all the first data contained in the batch, or a portion of the first data in the batch that is earlier or later in position may be selected, etc. This specification does not specifically limit this.

[0072] For example, assuming a batch contains four pieces of first data, representing student information for four students, with student IDs as the primary key, and a sampling rate of 50%, the data checker can randomly select two pieces of first data from the four pieces in the batch and obtain their primary key values, i.e., two student IDs (e.g., student ID 15 and student ID 40). Then, the data checker can use these two student IDs to perform a reverse lookup in the target system to find two pieces of second data containing the same two student IDs. The data checker can then verify these two pieces of first data against the corresponding two pieces of second data. For example, it can verify the first data with student ID 15 against the second data with the same student ID 15, and the first data with student ID 40 against the second data with the same student ID 40, to check if all fields in each piece of first data correspond one-to-one with the corresponding second data.

[0073] In one illustrated embodiment, the data checker can also select a subset of batches from all batches included in the Data batch component based on a preset sampling rate, wherein the proportion of this subset of batches in all batches can be equal to the sampling rate. For example, assuming a sampling rate of 50%, the data checker can select one batch from every two batches in the Data batch component; assuming a sampling rate of 75%, the data checker can select three batches from every four batches in the Data batch component, and so on. This specification does not specifically limit this. Furthermore, the data checker can perform the data verification process described above for each selected batch, which will not be elaborated further here.

[0074] In one illustrated embodiment, after verifying the first data and the second data, the data checker determines that the first data and the second data are completely identical, thus confirming a successful data verification. For example, if both the first data and the second data are student information, and all field values ​​in the first data and the second data, such as the name field, student ID field, and class field, are identical, then the data verification can be considered successful.

[0075] In one illustrated embodiment, after verifying the first data and the second data, if the data checker determines that there are at least some differences between the first data and the second data, it can determine that the verification of the first data and the second data has failed. For example, still taking student information as the case, if the student ID and class fields in the first data and the second data contain the same values, but the name field (i.e., the student's name) in the first data and the second data contains different values—for example, the name field in the first data contains "Zhang San," while the name field in the second data contains "Li San"—then it can be determined that the verification of the first data and the second data has failed.

[0076] In one illustrated embodiment, this application considers that due to possible technical malfunctions or accidents, the data checker may incorrectly determine that the originally identical first and second data are a failed check, resulting in incorrect data check results and affecting subsequent data management. Therefore, this application further designs a fault-tolerance mechanism for data check. The data checker can repeatedly check the failed first and second data within a preset tolerance window. If, after reaching the preset tolerance window, it is still determined that there is at least a partial difference between the first and second data, then it can be finally determined that the first and second data have failed to check.

[0077] In one illustrated embodiment, the preset tolerance window can be a preset verification time or a preset number of verifications, which is not specifically limited in this specification. Specifically, after the data verifier determines that there is at least a partial difference between the first data and the second data, it can verify the first data and the second data again. Further, if after the second verification it is still determined that there is at least a partial difference between the first data and the second data, the verification can continue, and so on, until the preset verification time or the preset number of verifications is reached, at which point the verification ends. Further, if after reaching the preset verification time or the preset number of verifications, it is still determined that there is at least a partial difference between the first data and the second data, then the verification of the first data and the second data can be ultimately determined to have failed; otherwise, if within the preset verification time or the preset number of verifications it is determined that the first data and the second data are completely identical, then the verification of the first data and the second data can be ultimately determined to have succeeded.

[0078] It should be understood that if the data checker cannot find the second data related to the primary key value of the first data in the target system (for example, it cannot find the second data with student ID 15), then the first data check can also be determined to have failed. In one illustrated embodiment, if the data checker is unable to find the second data related to the primary key value of the first data in the target system within a preset search time or number of searches, then the first data check can be determined to have failed.

[0079] In one illustrated embodiment, the data checker can also calculate a failure rate based on the number of first data items that have been checked and the number of first data items that failed to be checked. Specifically, the proportion of the number of first data items that failed to be checked to the total number of first data items that have been checked can be calculated, and this proportion can be determined as the failure rate.

[0080] Furthermore, the data checker can determine whether the calculated check failure rate is greater than a preset threshold. If the check failure rate is greater than the preset threshold, the current data integration can be stopped, that is, the synchronization of data from the source system to the target system can be stopped in time to prevent further damage and avoid amplifying data quality problems in the target system. In one illustrated embodiment, if the check failure rate is greater than the preset threshold, the data checker can send a signal to the data loader to notify the data loader to stop writing the first data in the target format contained in the batch to the target system.

[0081] In one illustrated embodiment, the user can set the preset threshold for the verification failure rate according to actual conditions and needs, such as 10%, 15%, or 35%, etc., which is not specifically limited in this specification. For example, if the preset threshold for the verification failure rate is set to 0, it means that the verification failure rate is ignored and does not affect data integration; or, the preset threshold can be set to 100% to mean that the verification failure rate is ignored, which is not specifically limited in this specification.

[0082] In one illustrated embodiment, after determining that the first data and the second data fail to match, the data checker can also store the failed first data, the second data, and the matching result to a preset database or to other systems, so that users can analyze and find data anomalies and perform corresponding data management operations on the target system to ensure the data integrity and accuracy of the target system, etc. This specification does not make specific limitations in this regard.

[0083] In one illustrated implementation, considering that data integration involves concurrent writes—for example, multiple data update operations from the source system are concurrently written to the target system—frequent data modifications can lead to inconsistencies in short-term verification, thereby increasing the verification failure rate. For instance, suppose the first data extracted from the source system is the account information of account A. If account A transfers funds to account B multiple times within a short period, the balance of account A will change frequently. This can easily cause the balance field in the currently retrieved second data to differ from that in the first data.

[0084] Based on this, this application also provides a data verification method that can periodically re-verify the data that failed to be verified stored in the aforementioned preset database, so as to further reduce the verification failure rate.

[0085] In one of the illustrated implementations, please refer to Figure 4 , Figure 4 This is a flowchart illustrating a data verification method provided in an exemplary embodiment. For example... Figure 4 As shown, the data checker can periodically retrieve the primary key values ​​of failed checks from a preset database based on a preset time interval. Furthermore, the data checker can use the retrieved primary key values ​​to perform reverse lookups in the source and target systems, retrieving source data related to the primary key value from the source system and target data related to the primary key value from the target system. Further, the data checker can verify the source and target data and perform corresponding data management on the target system based on the verification results.

[0086] In one illustrated embodiment, after verifying the source data and target data, if the data checker determines that the source data and target data are at least partially different, it can determine that the target data is indeed abnormal. Then, the target data in the target system can be modified so that the modified target data is the same as the source data, ensuring the data integrity and accuracy of the target system, thereby providing reliable data services to users.

[0087] In one illustrated embodiment, after verifying the source data and target data, if the data verifier determines that the source data and target data are completely identical, it can delete the data that originally failed to be verified and its verification result from the preset database. Alternatively, it can modify the verification result in the preset database, re-recording the data that originally failed to be verified as the data that was successfully verified, and so on. This specification does not specifically limit this. For example, after a period of time, the aforementioned account A no longer transfers funds to account B, and all previous balance changes of account A have been successfully written into the target system. At this time, the source data and target data corresponding to account A are completely identical. Therefore, the first data, second data, and their verification result corresponding to account A that originally failed to be verified can be deleted from the preset database, or their verification result can be modified to a successful verification, and so on. This specification does not specifically limit this.

[0088] In summary, this application adds data verification capabilities to ETL, implementing ETLV (Extract, Transform, Load, Verify). This application not only focuses on the data integration process but also emphasizes data consistency between the source and target systems. By performing data verification during or after data integration, the consistency between the target and source systems can be verified. This process involves not only data quality checks but also data auditing and verification, ensuring consistency in content, structure, and format. Data consistency verification not only improves data quality but also ensures data consistency and compliance, thereby increasing data credibility, helping users use data with confidence, and guaranteeing a positive user experience.

[0089] To reiterate, data consistency verification not only confirms the integrity of data during synchronization or migration but also helps identify potential data quality issues, thereby improving overall data governance and providing enterprises with a more reliable basis for decision-making. Ultimately, through efficient data integration and rigorous data verification mechanisms, more efficient data processing, more reliable data analysis, and more secure data management can be provided, offering strong support for data-driven decision-making and enabling enterprises to leverage data analysis to gain a greater strategic advantage in fierce market competition.

[0090] In one illustrated embodiment, Figure 2The data extractor, data batching component, data converter, data loader, and data checker shown can be implemented in software, hardware, or a combination of both. This specification does not impose any specific limitations on this.

[0091] In addition, it should be noted that, Figure 2 The structure of the data verification device 300 shown is merely illustrative. In some possible embodiments, the data verification device 300 may include a comparison... Figure 2 The number of more or fewer components shown is not specifically limited in this specification.

[0092] For example, if users do not need to perform real-time data verification during the data integration process, or if users only need to verify the data between the source system and the target system after data integration, this step can be omitted. Figure 2 The data loader in the middle, correspondingly, can have its data integration function implemented independently by other data integration devices. The data verification device 300 provided in this application can only implement the data verification function. In one illustrated embodiment, the data verification device 300 can periodically verify the data between the source system and the target system at certain time intervals to reduce system pressure. For example, the data extractor can periodically extract a certain amount of first data from the source database according to a preset period, organize one or more of the first data into a batch, and write the batch into the Data batch component. Further, the data extractor can send a signal to the data converter to convert the first data in the batch into first data in the target format supported by the target system. Further, the data converter can directly send a signal to the data verifier to execute the data verification process described above for the first data in the target format contained in the batch. For details, please refer to the above. Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0093] For example, if the data format in the target system is the same as that in the source system, and the data structure is relatively simple, then this step can be omitted. Figure 2 The data converter in the process. Correspondingly, after the data extractor writes the batch to the Data batch component, it can directly send a signal to the data loader, causing the data loader to write the first data contained in the batch to the target system, and so on. Subsequent processes can be referenced above. Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0094] For example, if users only need to perform data verification periodically, and the data format in the target system is the same as that in the source system, then this step can be omitted. Figure 2 The data loader and data converter are located within the data batch component. Correspondingly, after the data extractor writes the batch to the Data batch component, it can directly send information to the data checker, enabling the data checker to perform the data verification process described above on the first data contained in the batch. For details, please refer to the above. Figure 2-4 The description of the corresponding embodiments will not be repeated here.

[0095] For example, if users only need to perform data integration and do not need to perform data verification, this step can be omitted. Figure 2 The data checker in the batch. Correspondingly, the data loader can terminate the process after writing the first data in the target format contained in the batch to the target system.

[0096] In summary, users can choose to use ETLV, ETV, ELV, EV or ETL according to their actual situation and needs. This manual does not make any specific restrictions on this.

[0097] Please see Figure 5 , Figure 5 This is a flowchart illustrating a data verification method provided in an exemplary embodiment. This method can be applied to... Figure 1 or Figure 2 The system architecture shown can be specifically applied to Figure 1 or Figure 2 The data verification device 300 in the system architecture shown is an example. Figure 5 As shown, the method may specifically include the following steps S501-S504.

[0098] Step S501: Extract first data from the data contained in the source system. The first data contains multiple fields.

[0099] Step S502: Determine the primary key field among the plurality of fields included in the first data, and obtain the primary key value in the primary key field.

[0100] Step S503: Find the second data related to the primary key value from the data contained in the target system; wherein at least a portion of the data in the target system is data synchronized from the source system, and the at least a portion of the data includes the second data.

[0101] Step S504: Verify the first data and the second data, and perform corresponding data management on the target terminal system based on the verification result.

[0102] The specific implementation methods for steps S501-S504 above can be found in the above description. Figures 1-4 The description of the corresponding embodiments will not be repeated here.

[0103] Corresponding to the above-described method and process, embodiments of this specification also provide a data verification device. Please refer to... Figure 6 , Figure 6 This is a schematic diagram of a data verification device provided in an exemplary embodiment. The device 60 can be applied to… Figure 1 or Figure 2 The system architecture shown can be specifically applied to Figure 1 or Figure 2 The data verification device 300 in the system architecture shown is an example. Figure 6 As shown, the device 60 includes:

[0104] The source data extraction unit 601 is used to extract first data from the data contained in the source system, the first data containing multiple fields;

[0105] The primary key value acquisition unit 602 is used to determine the primary key field among the plurality of fields included in the first data, and to acquire the primary key value in the primary key field;

[0106] The target-end data extraction unit 603 is used to find second data related to the primary key value from the data contained in the target-end system; wherein at least a portion of the data in the target-end system is data synchronized from the source-end system, and the at least a portion of the data includes the second data;

[0107] The data verification unit 604 is used to verify the first data and the second data, and to perform corresponding data management on the target terminal system based on the verification result.

[0108] In one illustrated embodiment, the source-side data extraction unit 601 is specifically used for:

[0109] Multiple first data are extracted in batches from the source system;

[0110] The plurality of first data are divided into at least one batch, and each batch is assigned a sequence number; wherein each batch contains at least a portion of the first data from the plurality of first data.

[0111] According to the respective sequence number of the at least one batch, each batch in the at least one batch is sequentially written into the pre-configured data component.

[0112] In one illustrated embodiment, the primary key value acquisition unit 602 is specifically used for:

[0113] In the pre-configured data component, the primary key field of the first data contained in each batch is determined sequentially according to the batch number, and the primary key value in the primary key field of the first data is obtained.

[0114] In one illustrated embodiment, the primary key value acquisition unit 602 is specifically used for:

[0115] According to a preset sampling rate, a portion of the first data is randomly selected from all the first data contained in the batch; wherein, the proportion of the portion of the first data in all the first data is equal to the sampling rate;

[0116] Obtain the primary key value from the primary key field of the first data portion.

[0117] In one illustrated embodiment, the data verification unit 604 is specifically used for:

[0118] If the first data and the second data are compared, and there are at least some differences between the first data and the second data, then it is determined that the comparison between the first data and the second data has failed.

[0119] Store the first data, the second data, and their verification results in a preset database;

[0120] Based on the failed verification data stored in the preset database, corresponding data management is performed on the target system.

[0121] In one illustrated embodiment, the data verification unit 604 is specifically used for:

[0122] If there is at least a partial difference between the first data and the second data, then the first data and the second data are checked again. If it is still determined after the second check that there is at least a partial difference between the first data and the second data, then the first data and the second data are checked again, and so on, until the preset check time or the preset number of checks is reached.

[0123] If, after reaching the preset verification time or the preset number of verifications, it is still determined that there is at least a partial difference between the first data and the second data, then it is further determined that the verification of the first data and the second data has failed.

[0124] In one illustrated embodiment, the data verification unit 604 is specifically used for:

[0125] Based on a preset periodic duration, the primary key values ​​of the failed verification data are periodically retrieved from the preset database.

[0126] The source data related to the primary key value is retrieved from the source system, and the target data related to the primary key value is retrieved from the target system.

[0127] The source data is compared with the target data, and corresponding data management is performed on the target system based on the comparison result.

[0128] In one illustrated embodiment, the data verification unit 604 is specifically used for:

[0129] If the source data and the target data are checked, and the source data and the target data are at least partially different, the target data in the target terminal system is modified so that the modified target data is the same as the source data.

[0130] In one illustrated embodiment, the device 60 further includes a data modification unit 605, for:

[0131] If the source data is completely identical to the target data, then the data that failed to be verified is deleted from the preset database, or the data that failed to be verified is re-recorded as data that succeeded in the preset database.

[0132] In one illustrated embodiment, the data format of the data in the source system is different from the data format of the data in the target system; the data verification unit 604 is specifically used for:

[0133] Based on the target format of the data in the target terminal system, the first data is converted to obtain the first data in the target format; wherein, the data format of the second data is the target format;

[0134] The first data in the target format is compared with the second data in the target format.

[0135] In one illustrated embodiment, the device 60 further includes a data loading unit 606, configured to:

[0136] Write the first data in the target format into the target terminal system.

[0137] In one illustrated embodiment, the device 60 further includes a failure rate calculation unit 607, for:

[0138] Calculate the verification failure rate based on the number of first data that has been verified and the number of first data that failed to be verified.

[0139] Determine whether the verification failure rate is greater than a preset threshold. If the verification failure rate is greater than the preset threshold, stop the data synchronization between the source system and the target system.

[0140] The specific implementation process of the functions and roles of each unit in the aforementioned device 60 is detailed in the description of the above embodiments and will not be repeated here. It should be understood that the aforementioned device 60 can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor (CPU) of the device loading the corresponding computer program instructions into memory for execution. From a hardware perspective, in addition to the CPU and memory, the device typically includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards for implementing network communication functions.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of the solution described in this specification, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0142] The devices, units, and modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, in-vehicle computer, or any combination of these devices.

[0143] Corresponding to the above method embodiments, embodiments of this specification also provide a computing device. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. Figure 7 The computing device shown can be Figure 1 or Figure 2 The data verification device 300 in the system architecture shown is an example. Figure 7 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (e.g., a keyboard) and an output device 1005 (e.g., a display). The processor 1001, memory 1002, input device 1004, and output device 1005 can be connected via a bus or other means. Figure 7As shown, the memory 1002 includes a computer-readable storage medium 1003 storing a computer program executable by the processor 1001. The processor 1001 may be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiments. When the processor 1001 runs the stored computer program, it can execute various steps of the data verification method in the embodiments of this specification, including: extracting first data from data contained in the source system, the first data containing multiple fields; determining a primary key field among the multiple fields contained in the first data, and obtaining the primary key value in the primary key field; finding second data related to the primary key value from data contained in the target system; wherein at least a portion of the data in the target system is data synchronized from the source system, and the at least a portion of the data includes the second data; verifying the first data and the second data, and performing corresponding data management on the target system based on the verification result, etc.

[0144] For a detailed description of each step of the above data verification method, please refer to the previous content, which will not be repeated here.

[0145] Corresponding to the above method embodiments, embodiments of this specification also provide a computer-readable storage medium storing computer programs that, when run by a processor, execute the various steps of the data verification method in the embodiments of this specification. Please refer to the description of the above embodiments for details, which will not be repeated here.

[0146] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0147] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0148] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0149] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data.

[0150] Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A data collation method, characterized by, The method comprises: batch extracting a plurality of first data from a source end system, the first data comprising a plurality of fields; dividing the plurality of first data into at least one batch, and assigning a serial number to each batch; wherein each batch comprises at least part of the plurality of first data; sequentially writing each batch in the at least one batch into a preconfigured data component in order according to the serial number of the at least one batch; in the preconfigured data component, sequentially determining the primary key field of the first data contained in each batch according to the serial number of the batch, and obtaining the primary key value in the primary key field of the first data; finding second data related to the primary key value from data contained in a target end system; wherein at least part of the data in the target end system is data synchronized from the source end system, and the at least part of the data contains the second data; checking the first data and the second data, and performing corresponding data management on the target end system based on the checking result.

2. The method of claim 1, wherein, The method comprises: according to a preset sampling rate, randomly selecting part of the first data from all the first data contained in the batch; wherein the proportion of the part of the first data in the all the first data is equal to the sampling rate; obtaining the primary key value in the primary key field of the part of the first data.

3. The method of claim 1, wherein, The method comprises: checking the first data and the second data, and if there is at least part of the first data and the second data that is not the same, determining that the checking of the first data and the second data fails; storing the first data, the second data and the checking result thereof into a preset database; based on the checking failure data stored in the preset database, performing corresponding data management on the target end system.

4. The method of claim 3, wherein, The method comprises: if there is at least part of the first data and the second data that is not the same, rechecking the first data and the second data, and if it is still determined that there is at least part of the first data and the second data that is not the same after rechecking, continuing to check the first data and the second data, and so on until a preset checking time is reached or a preset checking number of times is reached; if it is still determined that there is at least part of the first data and the second data that is not the same after the preset checking time is reached or the preset checking number of times is reached, further determining that the checking of the first data and the second data fails.

5. The method of claim 3, wherein, The method comprises: based on a preset periodic length, periodically obtaining the primary key value of the checking failure data from the preset database; finding out source data related to the primary key value from the source system, and finding out target data related to the primary key value from the target system; checking the source data and the target data, and performing corresponding data management on the target system based on the checking result.

6. The method of claim 5, wherein, The checking the source data and the target data, and performing corresponding data management on the target system based on the checking result, comprises: checking the source data and the target data, and if there is at least partial difference between the source data and the target data, modifying the target data in the target system so that the modified target data is the same as the source data.

7. The method of claim 6, wherein, The method further comprises: if the source data and the target data are all the same, deleting the failed checking data from the preset database, or re-recording the failed checking data as successful checking data in the preset database.

8. The method of claim 3, wherein, The data format of the data in the source system is different from the data format of the data in the target system. The checking the first data and the second data comprises: performing format conversion processing on the first data based on the target format of the data in the target system to obtain target format first data; wherein the data format of the second data is the target format; checking the target format first data and the target format second data.

9. The method of claim 8, wherein, Before finding out the second data related to the primary key value from the data contained in the target system, the method further comprises: writing the target format first data into the target system.

10. The method of claim 9, wherein, The method further comprises: calculating a failure checking rate according to the number of the checked first data and the number of the failed checking first data; determining whether the failure checking rate is greater than a preset threshold, and if the failure checking rate is greater than the preset threshold, stopping the data synchronization between the source system and the target system.

11. A data collating apparatus, characterized by comprising: The device comprises: a source data extraction unit configured to extract a plurality of first data from a source system in batches, divide the plurality of first data into at least one batch, and assign a serial number to each batch; and write each batch in the at least one batch into a preconfigured data component in an order according to the serial number of the at least one batch; wherein the first data comprises a plurality of fields, and each batch comprises at least part of the plurality of first data; a primary key value acquisition unit configured to determine a primary key field of the first data contained in each batch in the preconfigured data component in an order according to the serial number of the batch, and acquire a primary key value in the primary key field of the first data; a target data extraction unit configured to find out second data related to the primary key value from data contained in a target system; wherein at least part of the data in the target system is data synchronized from the source system, and the second data is contained in the at least part of the data. A data checking unit is configured to check the first data and the second data, and perform corresponding data management on the target end system based on a checking result.

12. A computing device, comprising: The method comprises: a memory and a processor; the memory stores a computer program / instruction executable by the processor; the processor executes the computer program / instruction to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer program / instruction stored on the memory is executed by the processor to implement the method according to any one of claims 1-10.

14. A computer program product, characterised in that, The computer program product comprises the computer program / instruction executable by the processor to implement the method according to any one of claims 1-10.

Images