Data consistency checking method and device, electronic device and storage medium

Abstract

The present invention discloses a data consistency checking method and device, electronic device, and storage medium thereof, relating to the field of financial technology. The checking method comprises: obtaining lake data recorded in a target data lake and source data transmitted to the target data lake; when transmitting a source file corresponding to the source data, counting the total number of file records in the source file; checking whether the total number of file records is consistent with the number of loaded records to obtain a first check result; checking whether the number of source table records is consistent with the number of loaded records to obtain a second check result; and determining that the lake data and the source data are consistent when the first check result and the second check result indicate that both checks are consistent. The present invention solves the technical problem in related technologies that data inconsistencies can be easily caused by the inability to check for missing data in data collection and data transmission.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method for checking data consistency, a device thereof, an electronic device, and a storage medium. Background Art

[0002] A data lake is a large warehouse that stores a wide variety of raw data. It can retrieve raw data from an organization's multiple business systems and centrally store it in copies. As the foundational data support for the entire data center, efficient quality assurance is imperative. Therefore, the integrity of the data lake's data quality inspection mechanism will have a crucial impact on the data's usability.

[0003] In related technologies, data lake data quality assurance has the following two flaws:

[0004] (1) Guarantee effect: There are multiple technical links such as data collection, transmission and loading between the data lake and the business system, and it is necessary to ensure the data consistency between the data lake and the business system. The existing data consistency guarantee method is mainly based on comparing the data transmitted to the data lake, and cannot deeply explore the data of each source application, and cannot check the data missing in data collection and data transmission. If there is data omission during the data collection and transmission process, it will cause problems with the integrity and accuracy of the data in the data lake, resulting in data inconsistency. For example, an account in the source application is updated to be closed, but the data in the source application is not downloaded, and the data in the data lake has not been updated. Due to the data life cycle, when the source application cleans and deletes the relevant data, it will cause inconsistency problems in the upstream and downstream data. Such problems are difficult to analyze and trace later, and data repair is complicated.

[0005] (2) Cost efficiency: Existing methods for ensuring data consistency mainly use custom scripts to build inspection rules, which requires a large amount of R&D resources. In addition, the data governance work of the data lake includes the formulation of quality inspection rules and problem governance. Both types of work require a large amount of in-depth participation from users related to each business system (data source application), and the level of automation is low, which affects the efficiency of the data lake data governance work.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0007] The embodiments of the present invention provide a data consistency checking method and device, an electronic device and a storage medium, so as to at least solve the technical problem in the related art that data collection and data transmission are unable to check for missing data, which easily leads to data inconsistency.

[0008] According to one aspect of an embodiment of the present invention, a method for checking data consistency is provided, comprising: obtaining lake data recorded in a target data lake and source data transmitted to the target data lake, wherein the source data at least comprises: a source system table, in which the number of source table records collected by each data source application is pre-recorded, and the lake data at least comprises: a loading table, in which the number of loaded records successfully loaded into the source data is pre-recorded; when transmitting a source file corresponding to the source data, counting the total number of file records of the source file; checking whether the total number of file records is consistent with the number of loaded records to obtain a first check result; checking whether the number of source table records is consistent with the number of loaded records to obtain a second check result; and determining that the lake data is consistent with the source data when the first check result and the second check result indicate that both checks are consistent.

[0009] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, the method further includes: obtaining file information pre-stored in the metadata management platform, wherein the file information includes at least: metadata and the file format of a metafile storing the metadata; checking whether the file format of the metafile is consistent with the file format of the source file corresponding to the source data, and obtaining a third inspection result; if the third inspection result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data, determining that the inspection of the source file corresponding to the source data has passed; if the third inspection result indicates that the file format of the metafile is inconsistent with the file format of the source file corresponding to the source data, determining that the inspection of the source file corresponding to the source data has failed.

[0010] Optionally, after determining that the source file corresponding to the source data fails the check, it also includes: reading the error data in the source file that fails the check to obtain first error data; transmitting the first error data to the first data source application that generates the source file; and receiving the modified source file returned by the first data source application.

[0011] Optionally, after checking whether the total number of records in the file is consistent with the number of loaded records and obtaining a first check result, it also includes: if the first check result indicates that the total number of records in the file is inconsistent with the number of loaded records, determining that the check of the number of loaded records has failed; if the check of the number of loaded records has failed, reading the source data that failed to load and obtaining second error data; transmitting the second error data to the first data lake application, wherein the first data lake application corresponds one-to-one to the data source application, and the first data lake application is used to locate the failure factor of the loading failure; receiving the loading failure factor returned by the first data lake application; based on the loading failure factor, adjusting the loading method of the source data; and reloading the source data using the adjusted loading method.

[0012] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, it also includes: obtaining the historical source data transmitted to the target data lake within a historical time period; segmenting the historical time period according to a preset time length to obtain multiple historical time segments; based on the historical source data and the multiple historical time segments, calculating the average parameters of the recorded data files within the historical time period; calculating the data difference between the total number of records in the file and the average parameter; checking whether the data difference is within a preset threshold range to obtain a fourth inspection result.

[0013] Optionally, after checking whether the data difference is within a preset threshold range and obtaining a fourth check result, the method further includes: when the fourth check result indicates that the data difference is not within the preset threshold range, determining that abnormal data transmission fluctuations occur in the target data lake when receiving the source file; when abnormal data transmission fluctuations occur in the target data lake, reading erroneous data in the source file with abnormal transmission to obtain third error data; transmitting the third error data to a second data source application that generates the source file; and receiving the modified source file returned by the second data source application.

[0014] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, it also includes: reading reference data pre-stored in the metadata management platform, wherein the reference data at least includes: key field information; based on the key field information, counting the number of key records for each key field in the loading table; checking whether the number of key records is a preset reference value, and obtaining a fifth check result.

[0015] Optionally, after checking whether the number of key records is a preset reference value and obtaining the fifth check result, it also includes: when the fifth check result indicates that the number of key records is the preset reference value, determining that the primary key duplication status check of the data primary key after loading the source data has passed; when the fifth check result indicates that the number of key records is not the preset reference value, determining that the primary key duplication status check of the data primary key after loading the source data has failed.

[0016] Optionally, after checking whether the number of source table records is consistent with the number of loaded records and obtaining a second check result, the method further includes: if the second check result indicates that the number of source table records is consistent with the number of loaded records, determining that the loading operation to load the data source is successful; if the second check result indicates that the number of source table records is inconsistent with the number of loaded records, determining that the loading operation to load the data source has failed; if the loading operation to load the data source has failed, reading the source data that failed to load to obtain fourth error data; and transmitting the fourth error data to a third data source application, wherein the third data source application is used to locate the failure factor of the loading failure.

[0017] According to another aspect of an embodiment of the present invention, a data consistency checking device is also provided, including: an acquisition unit, used to acquire lake data recorded in a target data lake and source data transmitted to the target data lake, wherein the source data at least includes: a source system table, in which the number of source table records collected by each data source application is pre-recorded, and the lake data at least includes: a loading table, in which the number of loaded records successfully loaded into the source data is pre-recorded; a statistical unit, used to count the total number of file records of the source file when transmitting the source file corresponding to the source data; a first checking unit, used to check whether the total number of file records is consistent with the number of loaded records, and obtain a first check result; a second checking unit, used to check whether the number of source table records is consistent with the number of loaded records, and obtain a second check result; a determination unit, used to determine that the lake data is consistent with the source data when the first check result and the second check result indicate that both checks are consistent.

[0018] Optionally, the inspection device further includes: a first acquisition module, used to obtain file information pre-stored in the metadata management platform after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, wherein the file information at least includes: metadata and the file format of the metafile storing the metadata; a first inspection module, used to check whether the file format of the metafile is consistent with the file format of the source file corresponding to the source data, and obtain a third inspection result; a first determination module, used to determine that the inspection of the source file corresponding to the source data has passed if the third inspection result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data; a second determination module, used to determine that the inspection of the source file corresponding to the source data has failed if the third inspection result indicates that the file format of the metafile is inconsistent with the file format of the source file corresponding to the source data.

[0019] Optionally, the inspection device also includes: a first reading module, which is used to read the error data in the source file that failed the inspection after determining that the source file corresponding to the source data failed the inspection to obtain first error data; a first transmission module, which is used to transmit the first error data to the first data source application that generated the source file; and a first receiving module, which is used to receive the modified source file returned by the first data source application.

[0020] Optionally, the inspection device further includes: a third determination module for, after checking whether the total number of records in the file is consistent with the number of loaded records and obtaining a first inspection result, determining that the inspection of the number of loaded records has failed if the first inspection result indicates that the total number of records in the file is inconsistent with the number of loaded records; a second reading module for, if the inspection of the number of loaded records has failed, reading the source data that failed to be loaded and obtaining second error data; a second transmission module for transmitting the second error data to a first data lake application, wherein the first data lake application corresponds one-to-one to the data source application, and the first data lake application is used to locate the failure factor of the loading failure; a second receiving module for receiving the loading failure factor returned by the first data lake application; a first adjustment module for adjusting the loading method of the source data based on the loading failure factor; and a first loading module for reloading the source data using the adjusted loading method.

[0021] Optionally, the inspection device also includes: a second acquisition module, which is used to obtain the historical source data transmitted to the target data lake within a historical time period after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake; a first processing module, which is used to segment the historical time period according to a preset time length to obtain multiple historical time segments; a first calculation module, which is used to calculate the average parameters of the recorded data files within the historical time period based on the historical source data and the multiple historical time segments; a second calculation module, which is used to calculate the data difference between the total number of records in the file and the average parameter; and a second inspection module, which is used to check whether the data difference is within a preset threshold range to obtain a fourth inspection result.

[0022] Optionally, the inspection device further includes: a fourth determination module, configured to, after checking whether the data difference is within a preset threshold range and obtaining a fourth inspection result, determine that abnormal data transmission fluctuations occur in the target data lake when receiving the source file if the fourth inspection result indicates that the data difference is not within the preset threshold range; a third reading module, configured to, when abnormal data transmission fluctuations occur in the target data lake, read erroneous data in the source file with abnormal transmission to obtain third error data; a third transmission module, configured to transmit the third error data to a second data source application that generates the source file; and a third receiving module, configured to receive the modified source file returned by the second data source application.

[0023] Optionally, the inspection device also includes: a fourth reading module, used to read reference data pre-stored in the metadata management platform after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, wherein the reference data at least includes: key field information; a first statistics module, used to count the number of key records for each key field in the loading table based on the key field information; a third inspection module, used to check whether the number of key records is a preset reference value to obtain a fifth inspection result.

[0024] Optionally, the inspection device also includes: a fifth determination module for checking whether the number of key records is a preset reference value, and after obtaining a fifth inspection result, determining that the primary key duplication status check of the data primary key after loading the source data has passed if the fifth inspection result indicates that the number of key records is the preset reference value; and a sixth determination module for determining that the primary key duplication status check of the data primary key after loading the source data has failed if the fifth inspection result indicates that the number of key records is not the preset reference value.

[0025] Optionally, the inspection device further includes: a seventh determination module for, after checking whether the number of source table records is consistent with the number of loaded records and obtaining a second inspection result, determining that the loading operation to load the data source is successful if the second inspection result indicates that the number of source table records is consistent with the number of loaded records; an eighth determination module for, when the second inspection result indicates that the number of source table records is inconsistent with the number of loaded records, determining that the loading operation to load the data source has failed; a fifth reading module for, when the loading operation to load the data source has failed, reading the source data that failed to load and obtaining fourth error data; and a fourth transmission module for transmitting the fourth error data to a third data source application, wherein the third data source application is used to locate the failure factor of the loading failure.

[0026] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned data consistency checking method.

[0027] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned data consistency checking method.

[0028] In the present disclosure, the lake data recorded in the target data lake and the source data transmitted to the target data lake are obtained. When the source file corresponding to the source data is transmitted, the total number of file records of the source file is counted, and the total number of file records is checked to see if it is consistent with the number of loaded records, to obtain a first inspection result. The number of source table records is checked to see if it is consistent with the number of loaded records, to obtain a second inspection result. When the first inspection result and the second inspection result indicate that both inspections are consistent, it is determined that the lake data and the source data are consistent. In the present invention, the lake data recorded in the data lake can be inspected based on the source system tables in each data source application and the source data transmitted to the data lake. The data quality inspection of the data lake can be extended to each data source application, and data missing in data collection and data transmission can be inspected. A complete inspection of data consistency can be performed, effectively improving the inspection quality of data consistency inspection, thereby solving the technical problem in related technologies that data missing in data collection and data transmission cannot be inspected, which easily leads to data inconsistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 is a flow chart of an optional data consistency checking method according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of an optional data file format check according to an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of an optional check of the number of loaded records according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of an optional check on the fluctuation rate of the total number of records in a file according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of an optional duplicate check of primary keys of loaded table data according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of an optional consistency check between the number of source table records and the number of loaded records according to an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of an optional device for checking the consistency of a data lake and a data source according to an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram of an optional data consistency checking device according to an embodiment of the present invention;

[0038] Figure 9 The figure is a hardware structure block diagram of an electronic device (or mobile device) for a method for checking data consistency according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0042] Data Lake: A large warehouse that stores a variety of raw data from an organization, where the data can be accessed, processed, analyzed, and transmitted.

[0043] It should be noted that the data consistency checking method and device thereof in the present disclosure can be used in the field of financial technology when checking data consistency, and can also be used in any field other than the field of financial technology when checking data consistency. The present disclosure does not limit the application field of the data consistency checking method and device thereof.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0045] The following embodiments of the present invention can be applied to various systems / applications / devices for checking data consistency. The present invention addresses the problem that existing solutions cannot solve the problem of data omission during data collection and transmission, and cannot perform complete data consistency verification. It is no longer based solely on the data transmitted to the data lake by each data source application for comparison, but can implement verification based on the source system tables in each data source application, extending the data quality inspection of the data lake to the upstream system, and effectively improving the data consistency verification effect. The present invention involves data file format inspection, daily load record number inspection, source file record number volatility inspection, primary key duplication inspection, upstream and downstream consistency inspection based on source system tables, etc. The inspection types are diversified, which can effectively improve the accuracy of data consistency inspection and ensure that the data in the data lake is consistent with the data source data.

[0046] The present invention will be described in detail below with reference to various embodiments.

[0047] Example 1

[0048] According to an embodiment of the present invention, an embodiment of a method for checking data consistency is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Figure 1 FIG. 1 is a flow chart of an optional data consistency checking method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0050] Step S101: Obtain lake data recorded in the target data lake and source data transmitted to the target data lake. The source data includes at least a source system table, which pre-records the number of source table records collected by various data source applications. The lake data includes at least a loading table, which pre-records the number of successfully loaded source data records.

[0051] Step S102: When transmitting the source file corresponding to the source data, the total number of file records of the source file is counted.

[0052] Step S103: Check whether the total number of records in the file is consistent with the number of loaded records, and obtain a first check result.

[0053] Step S104 , checking whether the number of source table records is consistent with the number of loaded records, and obtaining a second check result.

[0054] Step S105: When the first check result and the second check result indicate that the checks are consistent, it is determined that the lake data and the source data are consistent.

[0055] Through the above steps, the lake data recorded in the target data lake and the source data transmitted to the target data lake can be obtained. When transmitting the source file corresponding to the source data, the total number of file records of the source file is counted, and the total number of file records is checked to see if it is consistent with the number of loaded records to obtain a first inspection result. The number of source table records is checked to see if it is consistent with the number of loaded records to obtain a second inspection result. When the first inspection result and the second inspection result indicate that both inspections are consistent, it is determined that the lake data and the source data are consistent. In an embodiment of the present invention, the lake data recorded in the data lake can be inspected based on the source system tables in each data source application and the source data transmitted to the data lake. The data quality inspection of the data lake can be extended to each data source application, and data missing in data collection and data transmission can be inspected. A complete inspection of data consistency can be performed, effectively improving the data consistency inspection effect, thereby solving the technical problem in the related technology that data missing in data collection and data transmission cannot be inspected, which easily leads to data inconsistency.

[0056] The embodiments of the present invention are described in detail below in conjunction with the above steps. The above steps can be applied to the inspection device.

[0057] Step S101: Obtain lake data recorded in the target data lake and source data transmitted to the target data lake. The source data includes at least a source system table, which pre-records the number of source table records collected by various data source applications. The lake data includes at least a loading table, which pre-records the number of successfully loaded source data records.

[0058] In an embodiment of the present invention, lake data recorded in a target data lake can be obtained. The lake data is data transmitted by a data source application according to a preset period (for example, one month) (before the data source application transmits data, the data source application can obtain the data to be transmitted from its own database and generate the obtained data as file data for transmission). The lake data includes a loading table, etc., and the loading table pre-records the number of loaded records of successful source data loading, as well as the source data obtained and transmitted to the target data lake (the source data is the data source application obtains the data to be transmitted from its own database and generates file data from the obtained data). The source data includes a source system table, etc., and the source system table pre-records the number of source table records collected by each data source application.

[0059] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, the method further includes: obtaining file information pre-stored in the metadata management platform, wherein the file information includes at least: metadata and the file format of the metafile storing the metadata; checking whether the file format of the metafile is consistent with the file format of the source file corresponding to the source data, and obtaining a third inspection result; if the third inspection result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data, determining that the inspection of the source file corresponding to the source data has passed; if the third inspection result indicates that the file format of the metafile is inconsistent with the file format of the source file corresponding to the source data, determining that the inspection of the source file corresponding to the source data has failed.

[0060] In an embodiment of the present invention, the file information in the metadata management platform can be used as reference data, and the data files regularly received by the data lake (i.e., the source data transmitted to the target data lake) can be used as the detected units to detect whether the actual file format of the files to be entered into the lake is consistent with the preset file format. If not, the governance process is entered until the subsequent inspection is passed.

[0061] In this embodiment, the file information pre-stored in the metadata management platform can be automatically obtained according to a customized frequency (which can be set according to actual conditions, for example, one month) (the file information includes: metadata and the file format of the metafile storing the metadata, etc., and the file format includes: the number of fields, the field length, etc.), and the file format of the metafile (the pre-set file format) is checked to see whether it is consistent with the file format of the source file corresponding to the source data, and a third inspection result is obtained. If the third inspection result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data, it is determined that the inspection of the source file corresponding to the source data has passed; otherwise, it is determined that the inspection of the source file corresponding to the source data has failed.

[0062] Optionally, after determining that the source file corresponding to the source data has failed the check, it also includes: reading the error data in the source file that failed the check to obtain first error data; transmitting the first error data to the first data source application that generates the source file; and receiving the modified source file returned by the first data source application.

[0063] In an embodiment of the present invention, after determining that the source file corresponding to the source data has failed the inspection, the error data that failed the inspection (i.e., the first error data, which is the error data in the source file that failed the inspection) is connected to the data quality management platform, and the data quality management platform feeds back the first error data to the data source application for modification (i.e., the first error data is transmitted to the first data source application that generates the source file). After the modification of the data source application is completed, the modified source file returned by the data source application is received, and the next round of inspection is continued in the cycle.

[0064] Figure 2 FIG. 1 is a schematic diagram of an optional data file format check according to an embodiment of the present invention. Figure 2 As shown, the following steps are included:

[0065] S201. The data lake regularly receives data files as the detection unit, automatically connects to the metadata management platform, and automatically obtains the latest file metadata information according to the customized frequency, including the number of fields, field length and other information.

[0066] S202: Perform a consistency check between the preset file format in the file metadata information and the actual file format of the file received by the data lake.

[0067] S203: If the check passes, the check item passes and the check ends.

[0068] S204: If the check fails, the erroneous data that fails the check will be connected to the data quality management platform.

[0069] S205: The data quality management platform returns the erroneous data to the data source application for modification.

[0070] S206: After the data source application is modified, the modified file information is circulated for the next round of inspection.

[0071] After completing the data file format check, you can continue to check the total number of records in the source file. Of course, you can also check other items in parallel. The order and parallel execution order of each inspection item are within the protection scope of the embodiments of this application.

[0072] Step S102: When transmitting the source file corresponding to the source data, the total number of file records of the source file is counted.

[0073] In an embodiment of the present invention, the data lake can periodically receive data files (i.e., transmit source files corresponding to source data) as reference data, and the table loaded into the data lake (i.e., the loaded table) as the detection unit, to detect whether the number of data table records (i.e., the total number of file records) is consistent with the number of data records in the file entering the lake (i.e., the number of loaded records). If they are inconsistent, the governance process will be entered until the subsequent inspection is passed. In this embodiment, the number of file records received by the data lake can be automatically counted first (i.e., when transmitting the source file corresponding to the source data, the total number of file records of the source file is counted).

[0074] Step S103: Check whether the total number of records in the file is consistent with the number of loaded records, and obtain a first check result.

[0075] In an embodiment of the present invention, the number of received file records (i.e., the total number of file records) is compared with the number of records successfully loaded into the data lake table (i.e., the number of loaded records) to obtain a first check result.

[0076] Optionally, after checking whether the total number of file records is consistent with the number of loaded records and obtaining the first check result, it also includes: when the first check result indicates that the total number of file records is inconsistent with the number of loaded records, determining that the check of the number of loaded records has failed; when the check of the number of loaded records has failed, reading the source data that failed to load and obtaining second error data; transmitting the second error data to the first data lake application, wherein the first data lake application corresponds one-to-one to the data source application, and the first data lake application is used to locate the failure factor of the loading failure; receiving the loading failure factor returned by the first data lake application; based on the loading failure factor, adjusting the loading method of the source data; and reloading the source data using the adjusted loading method.

[0077] In an embodiment of the present invention, if the first check result indicates that the total number of records in the file is consistent with the number of records loaded, it is determined that the check on the number of records loaded has passed. Otherwise, it is determined that the check on the number of records loaded has failed. If the check on the number of records loaded has failed, the error data that failed the check (i.e., second error data, which is the source data read and failed to load) is connected to the data quality management platform. The data quality management platform feeds the second error data back to the data lake application for problem analysis and modification (i.e., the second error data is transmitted to the first data lake application, which has a one-to-one correspondence with the data source application). The first data lake application is used to locate the failure factor of the load failure (i.e., find the specific factor of the load failure, such as the load failure caused by a file format problem) and return the found failure factor to the inspection device. After receiving the load failure factor returned by the first data lake application, the inspection device can adjust the loading method of the source data based on the load failure factor and reload the source data using the adjusted loading method until the check passes.

[0078] The following is a detailed description of the method for checking the number of loaded records. Figure 3 is a schematic diagram of an optional loading record number check according to an embodiment of the present invention, such as Figure 3 As shown, the following steps are included:

[0079] S301. Automatically count the number of file records received by the data lake application (the received file is a file regularly transmitted by the data source application).

[0080] S302: Perform a consistency check between the number of received file records and the number of lake table loaded records in the data lake application.

[0081] S303: If the check passes, the check item passes and the check ends.

[0082] S304: If the check fails, the erroneous data that fails the check will be connected to the data quality management platform.

[0083] S305: The data quality management platform returns the erroneous data to the data lake application for problem analysis and correction.

[0084] S306: Locate the problem in the data lake. Modify the source data based on the located problem, reload the source data, and continue the cycle of the next round of inspection with the number of records loaded into the modified lake table.

[0085] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, it also includes: obtaining the historical source data transmitted to the target data lake within the historical time period; segmenting the historical time period according to a preset time length to obtain multiple historical time segments; based on the historical source data and the multiple historical time segments, calculating the average parameters of the recorded data files within the historical time period; calculating the data difference between the total number of records in the file and the average parameter; checking whether the data difference is within the preset threshold range to obtain a fourth inspection result.

[0086] In an embodiment of the present invention, the historical data files regularly received by the data lake (i.e., the historical source data transmitted to the target data lake) can be used as reference data, and the file data currently received by the data lake is used as the detected unit to detect whether the data fluctuation of the files currently received by the data lake is reasonable (the data fluctuation is reasonable if it is within the set threshold) to eliminate the impact of data leakage caused by system abnormalities in the data source application. If it is unreasonable, it will enter the governance process until the subsequent inspection passes.

[0087] In this embodiment, the historical source data transmitted to the target data lake within the historical time period can be obtained, and the average parameters of the recorded data files within the historical time period can be calculated (the specific calculation method can be: the historical time period is segmented according to a preset time length (which can be set according to the specific situation) to obtain multiple historical time segments, and based on the historical source data and the multiple historical time segments, the average parameters of the recorded data files within the historical time period are calculated; the number of records with an average preset time length can also be calculated first, and the ratio of the number of records to a preset value (the preset value can be set according to the specific situation, for example, set to 2, indicating that the data can fluctuate by 50%) is represented as the average parameter), the data difference between the total number of records in the file and the average parameter is calculated, and the data difference is checked whether it is within the preset threshold range (which can be set according to the specific situation) to obtain the fourth inspection result.

[0088] Optionally, after checking whether the data difference is within a preset threshold range and obtaining a fourth check result, the method further includes: if the fourth check result indicates that the data difference is not within the preset threshold range, determining that abnormal data transmission fluctuations occur in the target data lake when receiving the source file; if abnormal data transmission fluctuations occur in the target data lake, reading erroneous data in the source file with abnormal transmission to obtain third error data; transmitting the third error data to a second data source application that generates the source file; and receiving the modified source file returned by the second data source application.

[0089] In an embodiment of the present invention, when the fourth check result indicates that the data difference is not within a preset threshold range, it is determined that the data transmission fluctuation of the target data lake when receiving the source file is abnormal, indicating that the check item has passed. Otherwise, it is determined that the data transmission fluctuation of the target data lake when receiving the source file is abnormal, indicating that the check item has failed. When the check item fails (that is, when the data transmission fluctuation of the target data lake is abnormal), the error data that fails the check (that is, the third error data, the third error data is the error data in the source file with abnormal transmission) is connected to the data quality management platform. The data quality management platform feeds back the third error data to the data source application for problem analysis and modification (that is, the third error data is transmitted to the second data source application that generates the source file). After the modification of the data source application is completed, the modified source file returned by the data source application is received, and the next round of inspection is continued in a cycle.

[0090] Figure 4 is a schematic diagram of an optional file total record number fluctuation rate check according to an embodiment of the present invention, such as Figure 4 As shown, the following steps are included:

[0091] S401: Using the historical data files received by the data lake as reference data and the latest received files of the data lake as the detected unit, the number of records of the received files of the data lake is automatically counted (the received files are files regularly transmitted by the data source application).

[0092] S402: Calculate the difference between the number of received file records and the moving average number of historical file records of the historical received data file, and check the difference with the threshold value defined by the threshold setter to see if the deviation is within the set threshold value.

[0093] S403: If it is within the threshold, the check is passed and the check ends.

[0094] S404: If the data is outside the threshold and fails the check, the erroneous data that fails the check will be connected to the data quality management platform.

[0095] S405: The data quality management platform returns the erroneous data to the data source application for problem analysis and modification.

[0096] S406: After the data source application is modified, the number of modified received file records is cycled through for the next round of checking.

[0097] Optionally, after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, it also includes: reading the reference data pre-stored in the metadata management platform, wherein the reference data at least includes: key field information; based on the key field information, counting the number of key records for each key field in the loading table; checking whether the number of key records is a preset reference value, and obtaining a fifth check result.

[0098] In an embodiment of the present invention, the file information in the metadata management platform and the data files regularly received by the data lake can be used as reference data, and the data loaded into the data lake table can be used as the detected unit to detect whether there are records with duplicate primary keys. If there are no related records, the check passes; if there are related records, the check fails and enters the governance process until the subsequent check passes.

[0099] In this embodiment, the reference data pre-stored in the metadata management platform can be automatically obtained according to a customized frequency (which can be set according to actual conditions, for example, one month) (the reference data at least includes: key field information, etc.), and the key field information is used as the primary key information of the data in the data lake table (i.e., the loading table). The number of key records for each key field in the loading table is counted, and it is checked whether the number of key records is a preset reference value (for example, 1 or 2) to obtain the fifth check result.

[0100] Optionally, after checking whether the number of key records is a preset reference value and obtaining the fifth check result, it also includes: when the fifth check result indicates that the number of key records is the preset reference value, determining that the primary key duplication status check of the data primary key after loading the source data has passed; when the fifth check result indicates that the number of key records is not the preset reference value, determining that the primary key duplication status check of the data primary key after loading the source data has failed.

[0101] In an embodiment of the present invention, when the fifth check result indicates that the number of key records is a preset reference value, it is determined that the primary key duplication status check of the data primary key after the source data is loaded has passed; otherwise, it is determined that the primary key duplication status check of the data primary key after the source data is loaded has failed. For example, when the number of key records is equal to the normal value 1, it indicates that the check item has passed; if the number of primary key records is greater than the normal value 1, it indicates that the check item has failed. Afterwards, the erroneous data that failed the check can be connected to the data quality management platform, and the data quality management platform will feed back the erroneous data to the data lake application for analysis and modification. The data lake application can use the data files regularly received by the data lake as reference data to analyze the cause of the error and feed back the cause of the error to the data source application for modification. After the data source application is modified, the cycle continues for the next round of inspections.

[0102] Figure 5FIG. 1 is a schematic diagram of an optional duplicate check of primary key of loaded table data according to an embodiment of the present invention. Figure 5 As shown, the following steps are included:

[0103] S501. Using the file metadata information in the metadata management platform and the data files regularly received by the data lake as reference data, and the data lake loading table as the detected unit, the system automatically connects to the metadata management platform and automatically obtains the latest file metadata information, including primary key information, at a customized frequency.

[0104] S502: According to the customized frequency, the primary key information obtained above is used as the primary key of the data lake loading table, and the number of primary key records loaded into the data lake loading table by the file is counted according to the primary key information (i.e., the number of records is counted according to the primary key information).

[0105] S503: Check whether the number of records is greater than 1. If the number of primary key records is equal to 1, the check passes and the check item ends.

[0106] S504. If the number of primary key records is greater than 1, the check fails, and the erroneous data that fails the check is connected to the data quality management platform.

[0107] S505: The data quality management platform returns the erroneous data to the data lake application and the data source application for analysis and modification.

[0108] S506: After the data source application is modified, the modified file information is circulated for the next round of inspection.

[0109] Step S104 , checking whether the number of source table records is consistent with the number of loaded records, and obtaining a second check result.

[0110] In an embodiment of the present invention, the data source system table can be used as reference data, and the table after the data lake is loaded (i.e., the loaded table) can be used as the detected unit to detect whether the number of loaded records is consistent with the number of source table records. If they are inconsistent, the governance process is entered until the subsequent inspection is passed (i.e., whether the number of source table records is consistent with the number of loaded records is checked to obtain the second inspection result).

[0111] Optionally, after checking whether the number of source table records is consistent with the number of loaded records and obtaining a second check result, it also includes: when the second check result indicates that the number of source table records is consistent with the number of loaded records, determining that the loading operation to load the data source is successful; when the second check result indicates that the number of source table records is inconsistent with the number of loaded records, determining that the loading operation to load the data source has failed; when the loading operation to load the data source has failed, reading the source data that failed to load to obtain fourth error data; transmitting the fourth error data to a third data source application, wherein the third data source application is used to locate the failure factor of the loading failure.

[0112] In an embodiment of the present invention, the data source application can automatically collect the number of source table records of the source system table at a customized frequency, generate record number information, and import the record number information of the data source application into the data lake. After counting the number of loaded records of the loaded table, it can be checked whether the number of source table records is consistent with the number of loaded records. If the second check result indicates that the number of source table records is consistent with the number of loaded records, it is determined that the loading operation of loading the data source is successful. Otherwise, it is determined that the loading operation of loading the data source fails, which means that the check item fails. The error data that fails the check (i.e., the fourth error data, which is the source data read and failed to load) is connected to the data quality management platform. The data quality management platform feeds back the fourth error data to the data source application for problem analysis and modification (i.e., the fourth error data is transmitted to the third data source application, and the third data source application is used to locate the failure factor of the loading failure). After the modification of the data source application is completed, the next round of inspection is continued in a cycle.

[0113] Figure 6 FIG. 1 is a schematic diagram of an optional consistency check between the number of source table records and the number of loaded records according to an embodiment of the present invention. Figure 6 As shown, the following steps are included:

[0114] S601. The data source application automatically collects the number of records in the data source table (the data source table generates a lake entry file) at a customized frequency, generates record number information, and imports the record number information into the data lake application.

[0115] S602: Automatically count the number of lake table records in the data lake table in the data lake application.

[0116] S603. Check whether the number of source table records in the record number information is consistent with the number of lake table records.

[0117] S604. If the check passes, the check item passes and the check ends; if the check fails, the erroneous data that fails the check is connected to the data quality management platform.

[0118] S605: The data quality management platform returns the erroneous data to the data source application for problem analysis and modification.

[0119] S606: After the data source application is modified, the modified data source table is circulated to perform the next round of inspection.

[0120] Step S105: When the first check result and the second check result indicate that the checks are consistent, it is determined that the lake data and the source data are consistent.

[0121] In an embodiment of the present invention, when at least two check results (including: the first check result and the second check result, etc.) indicate that the check has passed, it is determined that the lake data and the source data are consistent. The optimal way to pass the check is that all the above checks (including: data file format check, daily loaded record number check, source file record number fluctuation check, primary key duplication check, and upstream and downstream consistency check based on the source system table) are passed. If one of the checks fails, the erroneous data involved is modified through the data source application and / or the data lake application, and the check continues until the check passes.

[0122] In the embodiments of the present invention, the problem of missing data that cannot be checked and verified in the data collection and transmission links can be avoided. By performing checks based on the source system tables of each data source application, the data quality check of the data lake is extended to the upstream source application, effectively improving the data consistency check effect. It can also automatically collect metadata platform information and generate data consistency quality check scripts, saving a lot of manual maintenance resources for custom scripts, realizing automated quality comparison, realizing unattended comparison process and automatic generation of inconsistency problems, and directly connecting the inconsistency problems after automatic comparison to the data quality management platform for online communication and analysis, saving offline communication costs and improving the efficiency of data governance work.

[0123] Example 2

[0124] The present application is described below in conjunction with a preferred data consistency checking method.

[0125] Figure 7 is a schematic diagram of an optional data lake and data source consistency check quality inspection device according to an embodiment of the present invention, such as Figure 7 As shown, it includes: data source application, data lake application, metadata management platform, and inspection device. By inputting the detected information in the data source application and the data lake application into the inspection device, the inspection device checks the consistency between the data lake and the data source based on the reference information input by the metadata management platform, and feeds back the detection information to the data source application and the data lake application. The specific consistency check between the data lake and the data source mainly includes five checks: data file format check, daily loaded record number check, source file record number volatility check, primary key duplication check, and upstream and downstream consistency check based on the source system table. Among them,

[0126] Data file format check is used to check the format of files provided by upstream. Before the file is put into the lake, the consistency between the actual file format and the preset file format is detected. File checks include: file record length, number of fields, separators, line breaks, field length, etc.

[0127] Daily load record count check is used to check the consistency between the number of records in the source file and the number of records in the current data partition of the table after loading into the lake.

[0128] The source file record number fluctuation check is used to check whether the ratio of the number of records in the daily file to the recent historical average number of records and the preset value (i.e., the moving average) is within the preset threshold range.

[0129] Primary key duplication check is used to check the uniqueness of primary keys in the data lake based on the primary key fields of file information in the metadata management platform.

[0130] Based on the upstream and downstream consistency check of the source system table, the data source application is used to count the number of records according to the specific attributes of the data table and the preset logic, and regularly transmit the record number information to the data lake to complete the upstream and downstream data verification.

[0131] The present invention provides a device for checking the consistency between a data lake and a data source, which can achieve the following beneficial effects:

[0132] (1) Improved results: As existing solutions cannot solve the problem of data omissions during data collection and transmission, and cannot perform complete data consistency verification, this embodiment no longer only compares data transmitted to the data lake by each data source application, but can also verify data based on the source system tables in each data source application, extending the data quality inspection of the data lake to the upstream system, effectively improving the data consistency verification effect.

[0133] (2) Improved cost efficiency: It can automatically collect metadata platform information and generate data consistency quality check scripts, saving a lot of manual maintenance resources for custom scripts, realizing automated quality comparison, achieving unattended comparison process and automatic generation of inconsistency problems, and directly connecting the inconsistency problems after automatic comparison to the data quality management platform for online communication and analysis, saving offline communication costs and improving the efficiency of data governance work.

[0134] Example 3

[0135] A data consistency checking device provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment 1.

[0136] Figure 8 is a schematic diagram of an optional data consistency checking device according to an embodiment of the present invention, such as Figure 8 As shown, the inspection device may include: an acquisition unit 80, a statistical unit 81, a first inspection unit 82, a second inspection unit 83, and a determination unit 84, wherein:

[0137] An acquisition unit 80 is configured to acquire lake data recorded in the target data lake and source data transmitted to the target data lake. The source data includes at least a source system table, which pre-records the number of source table records collected by various data source applications. The lake data includes at least a loading table, which pre-records the number of successfully loaded source data records.

[0138] A counting unit 81 is used to count the total number of file records in the source file when the source file corresponding to the source data is transmitted;

[0139] A first checking unit 82 is used to check whether the total number of records in the file is consistent with the number of loaded records, and obtain a first checking result;

[0140] A second checking unit 83 is used to check whether the number of source table records is consistent with the number of loaded records, and obtain a second checking result;

[0141] The determining unit 84 is configured to determine that the lake data and the source data are consistent when the first check result and the second check result indicate that both checks are consistent.

[0142] The above-mentioned inspection device can obtain the lake data recorded in the target data lake and the source data transmitted to the target data lake through the acquisition unit 80, and count the total number of file records of the source file when transmitting the source file corresponding to the source data through the statistical unit 81, check whether the total number of file records is consistent with the number of loaded records through the first inspection unit 82, and obtain a first inspection result, and check whether the number of source table records is consistent with the number of loaded records through the second inspection unit 83 to obtain a second inspection result, and determine that the lake data and the source data are consistent when the first inspection result and the second inspection result indicate that the inspection are consistent. In an embodiment of the present invention, the lake data recorded in the data lake can be inspected based on the source system tables in each data source application and the source data transmitted to the data lake, and the data quality inspection of the data lake can be extended to each data source application, so as to realize the inspection of data missing in data collection and data transmission, and perform a complete inspection of data consistency, effectively improve the effect of data consistency inspection, and thus solve the technical problem in the related art that data missing in data collection and data transmission cannot be inspected, which easily causes data inconsistency.

[0143] Optionally, the inspection device also includes: a first acquisition module, used to obtain file information pre-stored in the metadata management platform after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, wherein the file information includes at least: metadata and the file format of the metafile storing the metadata; a first inspection module, used to check whether the file format of the metafile is consistent with the file format of the source file corresponding to the source data, and obtain a third inspection result; a first determination module, used to determine that the source file corresponding to the source data has passed the inspection when the third inspection result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data; a second determination module, used to determine that the source file corresponding to the source data has failed the inspection when the third inspection result indicates that the file format of the metafile is inconsistent with the file format of the source file corresponding to the source data.

[0144] Optionally, the inspection device also includes: a first reading module, which is used to read the error data in the source file that failed the inspection after determining that the source file corresponding to the source data has failed the inspection, to obtain first error data; a first transmission module, which is used to transmit the first error data to the first data source application that generates the source file; and a first receiving module, which is used to receive the modified source file returned by the first data source application.

[0145] Optionally, the inspection device also includes: a third determination module, which is used to check whether the total number of file records is consistent with the number of loaded records and obtain a first inspection result. If the first inspection result indicates that the total number of file records is inconsistent with the number of loaded records, determine that the inspection of the number of loaded records has failed; a second reading module, which is used to read the source data that failed to load and obtain second error data if the inspection of the number of loaded records fails; a second transmission module, which is used to transmit the second error data to the first data lake application, wherein the first data lake application corresponds one-to-one to the data source application, and the first data lake application is used to locate the failure factor of the loading failure; a second receiving module, which is used to receive the loading failure factor returned by the first data lake application; a first adjustment module, which is used to adjust the loading method of the source data based on the loading failure factor; and a first loading module, which is used to reload the source data using the adjusted loading method.

[0146] Optionally, the inspection device also includes: a second acquisition module, which is used to obtain the historical source data transmitted to the target data lake within the historical time period after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake; a first processing module, which is used to segment the historical time period according to a preset time length to obtain multiple historical time segments; a first calculation module, which is used to calculate the average parameters of the recorded data files within the historical time period based on the historical source data and multiple historical time segments; a second calculation module, which is used to calculate the data difference between the total number of records in the file and the average parameter; and a second inspection module, which is used to check whether the data difference is within a preset threshold range to obtain a fourth inspection result.

[0147] Optionally, the inspection device also includes: a fourth determination module, which is used to check whether the data difference is within a preset threshold range and obtain a fourth inspection result. If the fourth inspection result indicates that the data difference is not within the preset threshold range, determine that the data transmission fluctuation of the target data lake when receiving the source file is abnormal; a third reading module, which is used to read the error data in the source file with abnormal transmission when the data transmission fluctuation of the target data lake is abnormal, and obtain third error data; a third transmission module, which is used to transmit the third error data to the second data source application that generates the source file; and a third receiving module, which is used to receive the modified source file returned by the second data source application.

[0148] Optionally, the inspection device also includes: a fourth reading module, used to read the reference data pre-stored in the metadata management platform after obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, wherein the reference data at least includes: key field information; a first statistical module, used to count the number of key records for each key field in the loading table based on the key field information; a third inspection module, used to check whether the number of key records is a preset reference value to obtain a fifth inspection result.

[0149] Optionally, the inspection device also includes: a fifth determination module, which is used to check whether the number of key records is a preset reference value, and after obtaining the fifth inspection result, if the fifth inspection result indicates that the number of key records is the preset reference value, determine that the primary key duplication status check of the data primary key after loading the source data has passed; and a sixth determination module, which is used to determine that the primary key duplication status check of the data primary key after loading the source data has failed if the fifth inspection result indicates that the number of key records is not the preset reference value.

[0150] Optionally, the inspection device also includes: a seventh determination module for, after checking whether the number of source table records is consistent with the number of loaded records and obtaining a second inspection result, determining that the loading operation to load the data source is successful if the second inspection result indicates that the number of source table records is consistent with the number of loaded records; an eighth determination module for, when the second inspection result indicates that the number of source table records is inconsistent with the number of loaded records, determining that the loading operation to load the data source has failed; a fifth reading module for, when the loading operation to load the data source has failed, reading the source data that failed to load and obtaining fourth error data; and a fourth transmission module for transmitting the fourth error data to a third data source application, wherein the third data source application is used to locate the failure factor of the loading failure.

[0151] The above-mentioned inspection device may also include a processor and a memory. The above-mentioned acquisition unit 80, statistical unit 81, first inspection unit 82, second inspection unit 83, determination unit 84, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0152] The processor includes a kernel that retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters can be adjusted to determine that the lake data is consistent with the source data when both the first and second check results indicate consistency.

[0153] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0154] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialized program having the following method steps: obtaining lake data recorded in a target data lake and source data transmitted to the target data lake; when transmitting a source file corresponding to the source data, counting the total number of file records in the source file; checking whether the total number of file records is consistent with the number of loaded records to obtain a first check result; checking whether the number of source table records is consistent with the number of loaded records to obtain a second check result; and determining that the lake data and the source data are consistent when the first check result and the second check result indicate that both checks are consistent.

[0155] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned data consistency checking method.

[0156] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned data consistency checking method.

[0157] Figure 9 FIG. 1 is a hardware structure diagram of an electronic device (or mobile device) for a method for checking data consistency according to an embodiment of the present invention. Figure 9 As shown, the electronic device may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 9 More or fewer components than shown, or with Figure 9 Different configurations shown.

[0158] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0161] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0162] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0164] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for checking data consistency, characterized in that: include: Obtain lake data recorded in a target data lake and source data transmitted to the target data lake, wherein the source data at least includes: a source system table, in which the number of source table records collected by each data source application is pre-recorded; and the lake data at least includes: a loading table, in which the number of loaded records successfully loaded into the source data is pre-recorded; When transmitting a source file corresponding to the source data, counting the total number of file records of the source file; Check whether the total number of records in the file is consistent with the number of loaded records, and obtain a first check result; Check whether the number of records in the source table is consistent with the number of records in the loaded table, and obtain a second check result; Based on file information pre-stored in the metadata management platform, checking whether a file format of a metafile corresponding to the metadata in the file information is consistent with a file format of a source file corresponding to the source data, to obtain a third check result; Based on historical source data transmitted to the target data lake during a historical time period, checking a data transmission fluctuation rate of the target data lake when receiving the source file, to obtain a fourth inspection result, wherein the data transmission fluctuation rate refers to a value indicating a degree of change in the total number of records in the file relative to an average number of records in the historical time period; Based on the pre-stored file information, checking the number of key records in each key field in the load table to obtain a fifth checking result; In a case where at least two of the first check result, the second check result, the third check result, the fourth check result, and the fifth check result indicate that the checks are consistent, it is determined that the lake data is consistent with the source data.

2. The inspection method according to claim 1, characterized in that: After obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, the following steps are also included: If the third check result indicates that the file format of the metafile is consistent with the file format of the source file corresponding to the source data, determining that the source file corresponding to the source data has passed the check; If the third check result indicates that the file format of the metafile is inconsistent with the file format of the source file corresponding to the source data, it is determined that the source file corresponding to the source data has failed the check.

3. The inspection method according to claim 2, characterized in that: After determining that the source file corresponding to the source data fails the check, the method further includes: Reading error data in a source file that fails the check to obtain first error data; transmitting the first error data to a first data source application that generates the source file; Receive the modified source file returned by the first data source application.

4. The inspection method according to claim 1, characterized in that: After checking whether the total number of records in the file is consistent with the number of loaded records and obtaining a first check result, the method further includes: If the first check result indicates that the total number of records in the file is inconsistent with the number of loaded records, determining that the check of the number of loaded records has failed; If the check of the number of loaded records fails, reading the source data that failed to be loaded to obtain second error data; transmitting the second error data to a first data lake application, wherein the first data lake application corresponds one-to-one to the data source application, and the first data lake application is used to locate a failure factor of the loading failure; receiving a loading failure factor returned by the first data lake application; Adjusting the loading method of the source data based on the loading failure factor; The source data is reloaded using the adjusted loading method.

5. The inspection method according to claim 1, characterized in that: After obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, the following steps are also included: Obtain historical source data transmitted to the target data lake within a historical time period; Segmenting the historical time period according to a preset duration to obtain multiple historical time segments; Calculating an average parameter of the recorded data files within the historical time period based on the historical source data and the plurality of historical time segments; Calculate the data difference between the total number of records in the file and the average parameter; Check whether the data difference is within a preset threshold range to obtain a fourth check result.

6. The inspection method according to claim 5, characterized in that: After checking whether the data difference is within a preset threshold range and obtaining a fourth check result, the method further includes: If the fourth check result indicates that the data difference is not within a preset threshold range, determining that data transmission fluctuation of the target data lake when receiving the source file is abnormal; When abnormal data transmission fluctuation occurs in the target data lake, reading error data in the source file where the abnormal transmission occurs to obtain third error data; transmitting the third error data to a second data source application that generates the source file; Receive the modified source file returned by the second data source application.

7. The inspection method according to claim 1, characterized in that: After obtaining the lake data recorded in the target data lake and the source data transmitted to the target data lake, the following steps are also included: Reading reference data pre-stored in the metadata management platform, wherein the reference data at least includes: key field information; Based on the key field information, count the number of key records for each key field in the loading table; Check whether the number of key records is a preset reference value to obtain a fifth checking result.

8. The inspection method according to claim 7, characterized in that: After checking whether the number of key records is a preset reference value and obtaining a fifth check result, the method further includes: If the fifth check result indicates that the number of key records is the preset reference value, determining that the primary key duplication status check of the data primary key after loading the source data passes; When the fifth check result indicates that the number of key records is not the preset reference value, it is determined that the primary key duplication status check of the data primary key after the source data is loaded fails.

9. The inspection method according to claim 1, characterized in that: After checking whether the number of records in the source table is consistent with the number of records in the loaded table and obtaining a second check result, the method further includes: If the second check result indicates that the number of records in the source table is consistent with the number of records to be loaded, determining that the loading operation on the data source is successful; If the second check result indicates that the number of records in the source table is inconsistent with the number of records to be loaded, determining that the loading operation on the data source has failed; In the case where a loading operation on the data source fails, reading the source data that failed to be loaded to obtain fourth error data; The fourth error data is transmitted to a third data source application, wherein the third data source application is used to locate a failure factor of the loading failure.

10. A data consistency checking device, characterized in that: include: an acquisition unit, configured to acquire lake data recorded in a target data lake and source data transmitted to the target data lake, wherein the source data at least includes: a source system table, in which the number of source table records collected by each data source application is pre-recorded; and the lake data at least includes: a loading table, in which the number of load records successfully loaded into the source data is pre-recorded; a statistical unit, configured to count the total number of file records in the source file when transmitting the source file corresponding to the source data; A first checking unit, configured to check whether the total number of records in the file is consistent with the number of loaded records, and obtain a first checking result; The second checking unit is configured to check whether the number of records in the source table is consistent with the number of loaded records, thereby obtaining a second checking result; based on the file information pre-stored in the metadata management platform, checking whether the file format of the metafile corresponding to the metadata in the file information is consistent with the file format of the source file corresponding to the source data, thereby obtaining a third checking result; based on the historical source data transmitted to the target data lake within a historical time period, checking the data transmission volatility of the target data lake when receiving the source file, thereby obtaining a fourth checking result, wherein the data transmission volatility refers to the degree of change of the total number of records in the file relative to the average number of records within a historical time period; based on the pre-stored file information, checking the number of key records in each key field in the loaded table, thereby obtaining a fifth checking result; A determination unit is configured to determine that the lake data is consistent with the source data when at least two of the first check result, the second check result, the third check result, the fourth check result, and the fifth check result indicate that the checks are consistent.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data consistency checking method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: It includes one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data consistency checking method described in any one of claims 1 to 9.

Images