Data quality inspection method, device and equipment, program product and storage medium

Abstract

The invention discloses a data quality inspection method, device and equipment, a program product and a storage medium, and relates to the technical field of big data. The method comprises the following steps: receiving at least one to-be-detected report sent by at least one data source; determining at least one dimension attribute to which each to-be-detected report belongs according to the data source and the service type corresponding to each to-be-detected report and each predetermined dimension attribute; determining a sub-target rule template corresponding to the current dimension attribute according to the current dimension attribute and a predetermined rule template library; generating a target rule template of the current to-be-detected report according to the sub-target rule templates respectively corresponding to the dimension attributes to which the current to-be-detected report belongs; obtaining a data quality inspection result of the current to-be-detected report according to the target rule template corresponding to the current to-be-detected report; and determining target data according to the current to-be-detected report and the data quality inspection result of the current to-be-detected report, and sending the target data to a target platform.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular to a data quality inspection method, device, equipment, program product and storage medium. Background Art

[0002] In today's data-driven business environment, data quality inspection is becoming increasingly important. Data quality inspection helps verify and improve the accuracy, completeness, consistency, timeliness and reliability of data.

[0003] The data quality inspection methods currently used by enterprises are mostly to establish a rule base based on the business data of each subsidiary, and to perform quality inspection on the data according to the data quality inspection rules recorded in the rule base. However, in actual applications, there are correlations and differences between different subsidiaries or departments of an enterprise. For example, some business data of enterprises in the same province need to follow the same quality inspection rules, while some business data need to follow their own quality inspection rules. Developing data quality inspection rules separately for each subsidiary with differences, so as to conduct data quality inspection, is not only time-consuming and labor-intensive, but also not universal, leading to the problem of repeated investment in development. In addition, this method will ignore the correlation and differences between the data of each subsidiary, resulting in the quality inspection results of some data items not being consistent with the actual situation. Especially when faced with a large amount of data and a wide variety of data to be tested, this method is more prone to errors and cannot guarantee the efficiency of data quality inspection. Summary of the invention

[0004] The embodiments of the present invention provide a data quality inspection method, device, equipment, program product and storage medium, which are used to accurately perform data quality inspection on each data to be inspected and improve the efficiency of data quality inspection.

[0005] In a first aspect, an embodiment of the present invention provides a data quality inspection method, comprising:

[0006] Receiving at least one report to be detected sent by at least one data source;

[0007] Determine at least one dimensional attribute to which each report to be detected belongs according to a data source, a business type, and predetermined dimensional attributes corresponding to each report to be detected;

[0008] For each dimensional attribute of the at least one dimensional attribute, according to the current dimensional attribute and a predetermined rule template library, determine a sub-target rule template corresponding to the current dimensional attribute;

[0009] For each report to be detected in the at least one report to be detected, generating a target rule template for the current report to be detected according to the sub-target rule templates corresponding to each dimensional attribute to which the current report to be detected belongs;

[0010] Obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested;

[0011] The target data is determined according to the current report to be detected and the data quality inspection result of the current report to be detected, and the target data is sent to the target platform.

[0012] In a second aspect, an embodiment of the present invention further provides a data quality inspection device, including:

[0013] A data receiving module, used for receiving at least one report to be detected sent by at least one data source;

[0014] An attribute determination module, used to determine at least one dimension attribute to which each report to be detected belongs according to a data source, a business type and predetermined dimension attributes corresponding to each report to be detected;

[0015] A template determination module, configured to determine, for each dimensional attribute of the at least one dimensional attribute, a sub-target rule template corresponding to the current dimensional attribute according to the current dimensional attribute and a predetermined rule template library;

[0016] A template generation module, used for generating a target rule template for the current report to be detected according to the sub-target rule templates corresponding to the dimensional attributes of the current report to be detected for each report to be detected in the at least one report to be detected;

[0017] A result determination module is used to obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested;

[0018] The data sending module is used to determine the target data according to the current report to be detected and the data quality inspection result of the current report to be detected, and send the target data to the target platform.

[0019] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data quality inspection method as described in any one of the embodiments of the present invention is implemented.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data quality inspection method as described in any one of the embodiments of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a data quality inspection method as described in any one of the embodiments of the present invention.

[0022] In the embodiment of the present invention, the report to be tested is grouped according to the data source and business type of the report to be tested, and the attributes of each dimension to which the report to be tested belongs can be accurately determined through the grouping of the report to be tested, thereby improving the accuracy of the quality inspection rule template corresponding to each dimensional attribute determined according to the rule template library, further improving the matching degree between the quality inspection rule template and the report to be tested, and ensuring the accuracy and consistency of data quality inspection. Automatically performing data quality inspection on the report to be tested according to the data quality inspection template reduces the time and possibility of error in manual configuration, and improves the efficiency of data quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A first flow chart of a data quality inspection method provided by an embodiment of the present invention;

[0025] Figure 2 A flowchart of determining a sub-goal rule template corresponding to a current dimension attribute provided by an embodiment of the present invention;

[0026] Figure 3 A second flow chart of a data quality inspection method provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a data quality inspection device provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. The purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or must use the relevant content of the solution.

[0031] Figure 1 The first flow chart of a data quality inspection method provided by an embodiment of the present invention is used to accurately perform data quality inspection on each data to be inspected, thereby improving the efficiency of data quality inspection. The method can be executed by a data quality inspection device provided by an embodiment of the present invention, and the device can be implemented in software and / or hardware. The following embodiments will be described by taking the device integrated in an electronic device as an example, and the electronic device can be a computer device or a server, etc. Figure 1 , the method may specifically include the following steps:

[0032] Step 101: Receive at least one report to be detected sent by at least one data source.

[0033] The data source may be a business system that provides the data to be tested, and the report to be tested is a report provided by the data source that records the data generated by the business system. The data source in this solution may be various departments of an enterprise or a subsidiary of an enterprise in various regions.

[0034] Specifically, the data format and data type in the report to be tested of each data source are different, and each data source can provide at least one report to be tested. Enterprises can use their data quality inspection platform to perform anomaly detection on the data of each data source in real time or on a scheduled basis. Each data source can send the report to be tested to the data quality inspection platform in real time or on a scheduled basis according to the situation of its business system, and the data quality inspection platform can receive multiple reports to be tested sent by each data source.

[0035] Step 102: Determine at least one dimensional attribute to which each report to be detected belongs according to a data source, a business type, and predetermined dimensional attributes corresponding to each report to be detected.

[0036] Among them, the dimension attribute is used to indicate the data class or data characteristic to which the data in the report to be tested belongs. For example, a report to be tested includes information on business matters handled by staff of a certain department in the past month, including financial matters, consulting matters, and qualification information handling matters. It can be determined that the employee information such as the employee number, department, and rank of the staff in the report to be tested belongs to the personnel information attribute dimension, the information on the financial matters handled belongs to the financial information attribute dimension, the various consulting matters handled belong to the consulting information attribute dimension, and the qualification information matters handled belong to the qualification attribute dimension.

[0037] Specifically, different data sources correspond to different data processing standards for subsidiaries, so different data sources correspond to different data quality inspection rules. And the data quality inspection rules corresponding to data of different business types in the same data source are also different. Therefore, each report to be tested can be grouped according to the information of the data source of each report to be tested, and each group of reports to be tested in the data source dimension can be obtained. According to the business type of each report to be tested in each group of the data source dimension, the data reports to be tested belonging to the same business type are grouped again to obtain each group of reports to be tested in the business type dimension in each group of reports to be tested in the data source dimension.

[0038] Exemplarily, there are 6 data sources: data source a, data source b, data source c, data source d, data source e and data source f, where data source a, data source b and data source c are subsidiaries of province m, data source d and data source e are subsidiaries of province n, and data source f is a subsidiary of province p. Different provinces follow different data quality inspection rules and data processing methods, but the data quality inspection rules and data processing methods followed by the data reports to be tested in the same province are the same. Therefore, the reports to be tested of data source a, data source b and data source c can be grouped into one group, data source d and data source e can be grouped into one group, and data source f can be grouped into one group.

[0039] After obtaining each group of data sources, classify the data sources of each group according to the business type corresponding to each data table to be tested. Exemplarily, data source a includes 5 reports to be tested, namely report 1, report 2, report 3, report 4 and report 5. Report 1 records the loan information of enterprise 1, report 2 records the repayment information of enterprise 1, report 3 records the qualification registration information of enterprise 1, report 2 records the qualification registration information of enterprise 2, and report 5 records the human resources information of the subsidiary of data source a. It can be determined that report 1 and report 2 belong to the same business type, report 3 and report 4 belong to the same business type, and report 5 belongs to one business type. In this way, the report to be tested corresponding to each business type in each group of data sources is determined.

[0040] Since different data sources follow different data quality inspection standards, the data quality inspection rules followed by the reports to be tested of different business types in the same data source are also different. Therefore, it is necessary to analyze the dimensional attributes of each report to be tested according to the data source to which it belongs and the corresponding business type. Exemplarily, data source a and data source b belong to province s, and data source a includes report 1, report 2 and report 3. Data source b includes report 4, report 5 and report 6. Then report 1, report 2, report 3, report 4, report 5 and report 6 belong to a group in the data source dimension, denoted as S. Among them, report 1, report 2 and report 5 belong to the same business type K, report 3 and report 4 belong to a business type F, and report 6 belongs to a business type H. Further, according to the business type of each data source of each report to be tested, it is determined that report 1, report 2 and report 5 belong to group SK, report 3 and report 4 belong to group SF, and report 6 belongs to group SH.

[0041] In an optional implementation, after each report to be tested is grouped according to the data source and business type, the dimension attribute to which the data in the report to be tested of each group belongs is determined according to the predetermined dimensional attributes, and at least one dimensional attribute to which each report to be tested belongs is obtained. Exemplarily, report 1 records the borrowing information of enterprise 1, wherein the borrowing information includes enterprise information, information about relevant personnel responsible for the borrowing in the enterprise, and information about the amount of the borrowing, etc. Then it can be determined that the data related to the enterprise information in report 1 belongs to the enterprise information dimension, the relevant personnel information belongs to the personnel information dimension, and the loan amount information belongs to the financial information dimension. Further, it is determined that report 1 of group SK belongs to the personnel information dimension, the enterprise information dimension, and the financial information dimension.

[0042] By analyzing the data source and business type of the reports to be tested, we can accurately determine the dimensional attributes of the reports to be tested, thereby ensuring that each group of reports to be tested can undergo data quality inspection in accordance with its specific quality standards and requirements, laying the foundation for subsequently improving the accuracy of data quality inspection.

[0043] Step 103: for each dimensional attribute in at least one dimensional attribute, determine a sub-target rule template corresponding to the current dimensional attribute according to the current dimensional attribute and a predetermined rule template library.

[0044] The rule template library stores rule templates for various dimensional attributes under various data sources and business types. The rule template is a predefined data quality inspection framework, which is used to evaluate and ensure the accuracy, completeness and consistency of data. The rule templates corresponding to the same dimensional attributes of different groups of reports to be tested are also different. The current dimensional attribute is the dimensional attribute of the report to be tested determined based on the data source and business type, and the sub-target rule template is the rule template corresponding to the dimensional attribute of the report to be tested.

[0045] In an optional implementation, after determining all dimensional attributes to which the report to be detected belongs, all data source rule templates followed by the data source of the report to be detected are determined in the rule template library according to the data source of the report to be detected. The business rule template corresponding to the business type of the report to be detected is determined in all data source rule templates. The sub-target rule template corresponding to all dimensional attributes to which the report to be detected belongs is determined in all business rule templates. Exemplarily, report 1 is SK group (S province K business type), and report 1 belongs to 3 dimensional attributes, namely personnel information dimension, enterprise information dimension and financial information dimension. The data quality inspection platform can search for all S province rule templates corresponding to the S group in the rule template library, and search for all rule templates of the K business type in all S province rule templates. After finding all rule templates of the K business type, the sub-target rule templates corresponding to the personnel information dimension, enterprise information dimension and financial information dimension are searched in all rule templates of the K business type.

[0046] Specifically, if the rule template followed by the data source of the report to be tested cannot be found, or the rule template followed by the data source is found but the corresponding business rule template cannot be found in the data source rule template, or the business rule template is found but the sub-target rule template corresponding to one or more attribute dimensions of the report to be tested cannot be found in the business rule template. The data quality inspection platform can analyze the data in the report to be tested based on the rule template library, and determine all data quality inspection rules corresponding to the data identifier in the quality inspection rule library based on the data identifier of each data of each dimension attribute.

[0047] In an optional implementation, if the rule template followed by the data source of the report to be detected is not found in the rule template library, or the rule template followed by the data source is found, but the corresponding business rule template cannot be found in the data source rule template, the business rule template corresponding to the report to be detected can be found in all the data source rule templates in the rule template library. Determine the sub-goal rule templates corresponding to all the dimensional attributes of the report to be detected from these business rule templates. Similarly, if the data quality inspection platform does not find the sub-goal rule template corresponding to the current dimensional attribute in the business rule template, it can find the sub-goal rule template corresponding to the current dimensional attribute in other business rule templates or other data source rule templates. Since the sub-goal rule template determined in this case is not completely determined according to the data source and business type corresponding to the report to be detected, the sub-goal rule template determined in this case may not be accurate enough. In this solution, in order to improve the accuracy of the sub-goal rule template, a display page can be generated according to the sub-goal rule template, so that the user can adjust the sub-goal rule template generated by the data quality inspection platform according to the display page, and obtain a sub-goal rule template that is completely applicable to the dimensional attribute corresponding to the report to be detected.

[0048] By automatically determining the target rule template of the report to be tested based on the data source, business type and dimension attributes, the accuracy and consistency of data quality inspection are ensured, and the time and possibility of error in manual configuration are reduced. When no sub-target rule template is matched, the sub-target rule template applicable to the dimension attribute of the report to be tested can be generated through interaction with the user through the display page, which improves the flexibility of determining the data quality inspection template.

[0049] Step 104: for each report to be detected in at least one report to be detected, generate a target rule template for the current report to be detected according to the sub-target rule templates corresponding to the dimensional attributes of the current report to be detected.

[0050] Among them, the target rule template is used to perform data quality inspection on each data in the report to be tested, and obtain the data quality inspection result of the report to be tested. Specifically, the report to be tested includes data of at least one attribute dimension, and the sub-target rule templates corresponding to each attribute dimension are summarized to obtain the target rule template of the report to be tested. If the report to be tested only includes data of one attribute dimension, the sub-target rule template corresponding to the attribute dimension is determined as the target rule template of the report to be tested. This step can ensure that accurate data quality inspection can be performed on the data of each dimensional attribute of each report to be tested.

[0051] Step 105: Obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested.

[0052] Specifically, the target rule template includes data quality inspection rules for each dimension attribute, and each quality inspection rule includes at least one detection function. Substituting the data in the report to be tested into the corresponding detection function, the detection result of the data can be obtained. According to the detection result of each detection function, the data quality inspection result of the data of the report to be tested can be determined. In this solution, optionally, obtaining the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested includes the following steps A1-A2:

[0053] Step A1: According to a preset rule attribute function library, the detection function of each rule attribute in the target rule template of the current report to be detected and the execution order of each detection function are determined.

[0054] Among them, the rule attribute function library includes detection functions corresponding to each pre-set rule data, and the detection function is used to test the rules of specific data attributes (such as data type check and data value range check, etc.). In an optional implementation, after determining the target rule template, the data quality inspection platform searches the rule attribute function library for the detection function of the rule attribute corresponding to each dimension attribute in the target rule template. After obtaining all the detection functions of the report to be tested, the functions and logical dependencies of the detection functions are analyzed, and the execution order of each detection function is determined according to the analysis results. For example, an execution order framework can be automatically constructed based on the logical dependencies between the detection functions, and the execution order of each function can be determined based on the execution order framework.

[0055] Exemplarily, detection function 1 is responsible for checking the format of the data (such as whether the date format is correct), while detection function 2 relies on data in the correct format to further analyze the range of the data (such as whether the date is within a reasonable time range). It can be determined that detection function 1 is executed before detection function 2.

[0056] Step A2: Generate a rule adapter for the current report to be tested according to each test function and the execution order of each test function; perform data quality inspection on the current report to be tested through the rule adapter to obtain a data quality inspection result of the current report to be tested.

[0057] Among them, the rule adapter is a component in the rule quality inspection process. The rule adapter can organize the detection functions according to a certain logic and order to automatically check the quality of the data. The rule adapter can integrate each detection function and the execution order of the detection function into one unit, and automatically perform data quality inspection on the data of the report to be tested through the detection function according to the execution order of the detection function to obtain the data quality inspection result.

[0058] In an optional implementation, the rule adapter can import the data items in the report to be tested into the corresponding detection function in the adapter one by one. Each function returns a detection result, which indicates whether the data item meets the quality standard. After obtaining the detection results of all detection functions, the results of all detection functions are summarized to form the final data quality inspection result of the current report to be tested. The data quality inspection result of the report to be tested includes the quality inspection result of each data, as well as the reasons why the quality inspection result is a quality inspection failure, etc.

[0059] Through the pre-set rule attribute function library, the detection function and its execution order in the target rule template can be quickly determined, ensuring the logic and accuracy of data quality inspection. The rule adapter improves the speed of data quality inspection, realizes the automation of data quality inspection, and improves the efficiency and accuracy of data quality inspection.

[0060] Step 106: Determine the target data according to the current report to be tested and the data quality inspection result of the current report to be tested, and send the target data to the target platform.

[0061] Wherein, the target data is the data in the report to be tested, and the data quality inspection result is the data of successful quality inspection. The target platform can be a corporate headquarters. The target platform can receive the target data of each data source sent by the data quality inspection platform. In an optional implementation, after obtaining the data quality inspection result of the report to be tested, all the data of successful quality inspection are screened out, and the screened data are formatted or converted as necessary to meet the target data receiving requirements of the target platform. According to the technical architecture supported by the target platform, a suitable data transmission method is selected to send the target data to the target platform. After receiving the target data, the target platform can summarize the target data with the existing data on the platform. The summarized data is sorted so that the target data can be used efficiently in the future. By sending the target data to the target platform, the target platform can receive high-quality data, which provides a solid foundation for subsequent data application and analysis.

[0062] The technical solution of this embodiment receives at least one report to be detected sent by at least one data source; according to the data source, business type and predetermined dimensional attributes corresponding to each report to be detected, at least one dimensional attribute to which each report to be detected belongs is determined; for each dimensional attribute in at least one dimensional attribute, according to the current dimensional attribute and the predetermined rule template library, the sub-target rule template corresponding to the current dimensional attribute is determined; for each report to be detected in at least one report to be detected, according to the sub-target rule templates corresponding to each dimensional attribute to which the current report to be detected belongs, the target rule template of the current report to be detected is generated; according to the target rule template corresponding to the current report to be detected, the data quality inspection result of the current report to be detected is obtained; according to the data quality inspection results of the current report to be detected and the current report to be detected, the target data is determined, and the target data is sent to the target platform. The technical solution of this embodiment is to group the reports to be detected according to the data source and business type of the reports to be detected, and the dimensional attributes to which the reports to be detected belong can be accurately determined by the grouping of the reports to be detected, which improves the accuracy of the quality inspection rule template corresponding to each dimensional attribute determined according to the rule template library, further improves the matching degree between the quality inspection rule template and the report to be detected, and ensures the accuracy and consistency of data quality inspection. Automatically perform data quality inspection on the test reports according to the data quality inspection template, reducing the time and possibility of errors in manual configuration and improving the efficiency of data quality inspection.

[0063] Figure 2 The flowchart of determining the sub-goal rule template corresponding to the current dimension attribute provided by the embodiment of the present invention is a refinement of the above embodiment. The specific method can be as follows Figure 2 As shown, the method may include the following steps:

[0064] Step 201 : for each report to be detected, according to the current data source of the current report to be detected, a data source rule template sub-library corresponding to the current data source is searched in the rule template library.

[0065] Among them, the data source rule template sub-library includes rule templates corresponding to various types of data sources. The rule template library stores data quality inspection rule templates for various data sources. Specifically, different types of data sources correspond to different data source rule templates. For example, the data sources in different provinces follow different rule templates. After obtaining each report to be tested, the data quality inspection platform determines the data source corresponding to each report to be tested, and groups each report to be tested according to the data source dimension according to the predetermined data source grouping standard. After obtaining each data source grouping, find the rule template sub-library corresponding to each group of data sources in the rule template library.

[0066] Determining the data source rule template sub-library corresponding to the data source can ensure that the data quality inspection rules used subsequently are consistent with the specific characteristics of the data source, thereby improving the accuracy and relevance of data quality inspection.

[0067] Step 202: Search all business rule templates corresponding to the current business type in the data source rule template sub-library according to the current business type corresponding to the current report to be detected, and search for the sub-target rule template corresponding to the current dimension attribute in all business rule templates corresponding to the current business type.

[0068] Among them, the sub-target rule template is a rule template corresponding to the dimension attribute to which the report to be tested belongs. The business type corresponding to the report to be tested is the business type corresponding to the data in the report to be tested. For example, the data in report 1 is the data generated when the enterprise is subject to qualification review, and the business type corresponding to report 1 is the enterprise qualification type. The data in report 2 is the financial data of the enterprise, and the business type corresponding to report 2 is the financial type. The dimension attribute is used to indicate the data class or data characteristic to which the data in the report to be tested belongs. Different data in a report to be tested may belong to different dimension attributes. For example, report 1 includes basic information of the enterprise, certificate information, and personnel information of each department. The basic information of the enterprise corresponds to the quality inspection rules of the enterprise data attributes, such as the quality inspection rules of the enterprise number. The certificate information corresponds to the quality inspection rules of the certificate attributes, such as the format quality inspection rules of the certificate photo. The personnel information of each department corresponds to the quality inspection rules of the personnel information attributes, such as the quality inspection rules of the personnel code and the department.

[0069] Specifically, the data quality inspection rules followed by the reports to be tested of different business types in the same data source are different. For example, the data in the reports of enterprise qualification type and financial type are different, and the corresponding data quality inspection rules are also different. After obtaining the data source rule template sub-library of the report to be tested, search for the business rule template corresponding to all the business rules of the report to be tested in the data source rule template sub-library. After obtaining all the business rule templates, search for the sub-target rule template that matches the various dimensional attributes to which the current report to be tested belongs in all the business rule templates. Exemplarily, after determining the business rule template corresponding to report 1, search for the quality inspection rules of certificate attributes, the quality inspection rules of enterprise data attributes, and the quality inspection rules of personnel information attributes in its business rule template, so as to obtain the sub-target rule template corresponding to the various dimensional attributes to which report 1 belongs.

[0070] By selecting appropriate rule templates based on business types and dimension attributes, the accuracy and relevance of data quality checks can be improved. This enables the data quality inspection platform to equip each business type and its related dimension attributes with the most appropriate data quality inspection rules, thus optimizing the efficiency and effectiveness of the entire data quality review.

[0071] Step 203: If the data source rule template sub-library cannot be found, or the business rule template cannot be found, or the sub-target rule template corresponding to the current dimension attribute cannot be found, a candidate rule template for the current dimension attribute to which the current report to be tested belongs is generated based on the data identifier and the quality inspection rule library, and a rule display page is generated based on the candidate rule template.

[0072] Among them, data identifiers are used to mark and distinguish different data. The quality inspection rule library stores all quality inspection rules corresponding to each data. The rule template library stores templates composed of various rules. Unlike the rule template library, the quality inspection rule library does not distinguish between data sources or business types, etc. Therefore, a piece of data can find multiple quality inspection rules corresponding to different data sources and different business types in the quality inspection rule library. The candidate rule template is automatically generated by the data quality inspection platform based on the quality inspection rule library, and corresponds to the various dimensional attributes of the report to be tested. Since the candidate rule template is generated based on the quality inspection rule library, the data quality inspection rules for each dimensional attribute in the candidate rule template may not be accurate enough and are not completely applicable to the current report to be tested. The display page includes the various data quality inspection rules corresponding to each data in the candidate rule template, and users can adjust the candidate rule template through the display page.

[0073] Specifically, if the data source rule template sub-library of the report to be tested cannot be found, or the corresponding business rule template cannot be found in the data source rule template sub-library, or the business rule template is found but the sub-target rule template corresponding to one or more attribute dimensions of the report to be tested cannot be found in the business rule template. The data quality inspection platform can analyze the data in the report to be tested according to the rule template library, and generate candidate rule templates for the current dimension attributes according to the data identifier of each data of the current dimension attribute and the quality inspection rule library. In this scheme, optionally, the candidate rule templates for the dimension attributes to which the current report to be tested belongs are generated according to the data identifier and the quality inspection rule library, including: for each data identifier, searching the quality inspection rule library for at least one identification rule corresponding to the current data identifier; generating candidate rule templates for the current dimension attributes according to all data identifiers and at least one identification rule corresponding to all data identifiers.

[0074] The identification rule is a data quality inspection rule corresponding to the data identifier in the quality inspection rule library, and one data identifier corresponds to at least one identification rule. Specifically, after determining that a candidate rule template needs to be generated, the data identifiers of all data in the report to be tested are extracted, and at least one identification rule corresponding to the data identifier is searched in the quality inspection rule library. All data identifiers of the current dimension attribute and the corresponding identification rules are combined to generate a candidate rule template for the current dimension attribute.

[0075] By matching at least one identification rule for each data identifier, the corresponding quality inspection rule is automatically searched for each data identifier, and a candidate rule template is generated, which reduces the time and possibility of errors in manual configuration and improves the efficiency of data quality inspection.

[0076] Step 204: If a rule operation instruction sent by the user through the rule display page is received within a preset time period, the identification rule in the rule display page is modified according to the rule operation instruction.

[0077] The rule operation instructions include adding rules, deleting rules and changing rule status, and the rule status is enabled or disabled. The rule display page includes candidate rule templates, and the candidate rule templates include data identification and identification rules. Exemplarily, the candidate rule templates are shown in Table 1 below:

[0078] Data name Data Identification Identification Rule 1 Identification Rule 2 Identification Rule 3 Identification Rule 4 Identification Rule 5 Enterprise Code 1a Non-empty rules Length rule Format rule 1 Format rule 2 Company Name 1b Non-empty rules Length rule Format rule 1 Format rule 2 Format rule 3 Number of departments 1c Non-empty rules Length rule Format rule 1

[0079] Table 1

[0080] As shown in Table 1, the data identifier 1a of the enterprise code corresponds to 4 identification rules, the data identifier 1b of the enterprise name corresponds to 5 identification rules, and the data identifier of the number of departments corresponds to 3 identification rules. The user can send a rule operation instruction through the rule display interface, add an identification rule of a certain data identifier, or delete an identification rule of a certain data identifier, or modify the identification rule status of a certain data identifier through the rule operation instruction. For example, the enterprise code in Table 1 includes 4 identification rules. After viewing the content of each rule, the user determines that identification rule 4 is not applicable to the report to be detected, and the user can delete identification rule 4 of the enterprise code. In an optional implementation, each rule in the candidate rule template is in an enabled state. If the user determines that a certain identification rule does not need to be executed according to his own needs, but does not need to delete the identification rule (just not executed this time), the user can modify the status of the identification rule to a disabled state through the rule display page.

[0081] In this scheme, optionally, the identification rules in the rule display page are modified according to the rule operation instruction, including: when the rule operation instruction is to add a rule, according to the data identifier corresponding to the operation instruction, all addable rule items corresponding to the data identifier are determined; and the addable rule items are displayed to the user so that the user can add rules in the data identifier corresponding to the operation instruction according to the addable rule items.

[0082] Among them, the addable rule items are rules in the quality inspection rule base that have no obvious association with the data identifier but no contradiction. That is, the rules that are not included in the candidate rule template, but can be added to the corresponding data identifier. For example, the rules for user names in the candidate rule template include non-empty rules and length rules, and the quality inspection rule base includes content format rules for corporate names, which can also be used to specify user names, that is, the rule is an addable rule item corresponding to the user name identifier. Specifically, when the rule operation instruction is to add a rule, all addable rule items are determined according to the data identifier corresponding to the operation instruction, and the addable rule items are displayed to the user in the rule display page. The user can select one or more items from the addable rule items to add to the identification rule corresponding to the data identifier. Furthermore, when the rule operation instruction is to delete a rule, the identification rule corresponding to the rule operation instruction is deleted. When the rule operation instruction is to change the rule status, the status of the corresponding identification rule is modified according to the rule operation instruction.

[0083] By providing users with the option to add rule items, users can customize data identification rules according to specific needs, thereby improving data quality and accuracy. By precisely controlling the addition, deletion and status change of data rules, data quality inspection is flexible, accurate and user-friendly.

[0084] Step 205: Determine the sub-target rule template of the current dimension attribute according to the modified rule display page.

[0085] Among them, the sub-target rule template is the rule template corresponding to the dimension attribute to which the report to be tested belongs. In an optional implementation, after the user completes the modification of the candidate rule template through the rule display page, the data quality inspection platform obtains the candidate rule template of the current dimension attribute modified by the user through the rule display page, and determines the modified candidate rule template as the sub-target rule template of the current dimension attribute.

[0086] Step 206: If no rule operation instruction is received from the user through the rule display page within the preset time, or a confirmation instruction is received from the user through the rule display page, the candidate rule template of the current dimension attribute is determined as the sub-target rule template of the current dimension attribute.

[0087] Specifically, if no rule operation instruction is received from the user through the rule display page within the preset time, or a confirmation instruction is received from the user through the rule display page, indicating that the user does not need to modify the candidate rule template, the candidate rule template of the current dimension attribute is determined as the sub-target rule template of the current dimension attribute. This method allows the rule template determination process to be automatically promoted, thereby improving the efficiency of rule template determination.

[0088] Step 207: Store the sub-target rule template corresponding to the current dimension attribute generated according to the quality inspection rule library into the rule template library.

[0089] After the sub-goal rule template is generated, it is stored in the rule template library, so that subsequent users can directly use the sub-goal rule template to perform data quality inspection on the corresponding table to be inspected. This method ensures that the generated rule template can be effectively saved and reused to support continuous data quality inspection work.

[0090] The technical solution of this embodiment is to search the rule template sub-library of the current dimension attribute in the rule template library. If the rule template sub-library is found, the sub-target rule template corresponding to the business type to which the current dimension attribute belongs is searched in the rule template sub-library. If the rule template sub-library or the sub-target rule template corresponding to the business type to which the current dimension attribute belongs cannot be found, for each report to be detected, all data identifiers of the data belonging to the current dimension attribute in the current report to be detected are extracted according to the data source of the current report to be detected. A candidate rule template of the current dimension attribute to which the current report to be detected belongs is generated according to the data identifier and the quality inspection rule library, and a rule display page is generated according to the candidate rule template. If a rule operation instruction sent by the user through the rule display page is received within a preset time, the identification rule in the rule display page is modified according to the rule operation instruction; the rule operation instruction includes adding rules, deleting rules and changing rule status. The sub-target rule template of the current dimension attribute is determined according to the modified rule display page. If the rule operation instruction sent by the user through the rule display page is not received within the preset time, or a confirmation instruction sent by the user through the rule display page is received, the candidate rule template of the current dimension attribute is determined as the sub-target rule template of the current dimension attribute. The sub-target rule template corresponding to the current dimension attribute generated according to the quality inspection rule library is stored in the rule template library. The technical solution of this embodiment automatically determines the target rule template of the report to be tested according to the data source, business type and dimension attribute, which can ensure that the data quality inspection rules used subsequently are consistent with the specific characteristics of the data source and business type, and improve the accuracy and relevance of data quality inspection. The data quality inspection platform can be equipped with the most appropriate data quality inspection rules for each business type and its related dimension attributes, thereby improving the accuracy between data. When the sub-target rule template cannot be found, a candidate rule template can be generated according to the quality inspection rule library, and the rules of the candidate rule template can be adjusted through interaction with the user to obtain a quality inspection rule template suitable for the report to be tested. The efficiency, accuracy and user-friendliness of data quality inspection are achieved.

[0091] Figure 3 The second flow chart of a data quality inspection method provided by an embodiment of the present invention is a refinement of the above embodiment. The specific method can be as follows Figure 3 As shown, the method may include the following steps:

[0092] Step 301: Receive at least one report to be detected sent by at least one data source, and determine at least one dimensional attribute to which each report to be detected belongs according to the data source, business type and predetermined dimensional attributes corresponding to each report to be detected.

[0093] Step 302: for each dimensional attribute in at least one dimensional attribute, determine a sub-target rule template corresponding to the current dimensional attribute according to the current dimensional attribute and a predetermined rule template library.

[0094] Step 303: for each report to be detected in at least one report to be detected, generate a target rule template for the current report to be detected according to the sub-target rule templates corresponding to the dimensional attributes of the current report to be detected.

[0095] Step 304: obtain the data quality inspection result of the current report to be inspected according to the target rule template corresponding to the current report to be inspected, and generate a quality inspection success detailed table for the successfully inspected data according to all the data quality inspection results in the current report to be inspected.

[0096] Among them, the quality inspection result is a success or failure. After obtaining the target rule template corresponding to the current report to be tested, the data in the current report to be tested is quality inspected according to the target rule template, and the data quality inspection result is generated. If the quality inspection result is successful, it is determined that the data has no abnormalities. If the quality inspection result is a failure, it is determined that the data has abnormalities. After determining the data quality inspection results of all data in the current report to be tested, all data items with successful quality inspection results are screened out, and a quality inspection success detailed table including information such as data value, data source, and detection time is generated based on these data items, so as to provide a reliable basis for subsequent data use and analysis.

[0097] Step 305: determine the quality inspection success detailed list as target data, and send the target data to a predefined reporting data pool, so that the reporting data pool sends the quality inspection success detailed list to the target platform according to a preset period.

[0098] Among them, the reporting data pool is a temporary storage device for collecting, organizing and temporarily storing data to be distributed to different platforms. The preset cycle is a cycle predetermined according to specific needs and the actual operating environment. After obtaining the quality inspection success list, the quality inspection success list is determined as the target data, and the target data is sent to the pre-defined reporting data pool. After receiving the quality inspection success list, the reporting data pool stores the quality inspection success list and sends the quality inspection success list to the target platform according to the preset cycle.

[0099] By submitting data to the data pool, data can be processed and forwarded to the target platform in batches, rather than processing each data item individually, which reduces the data processing pressure on the target platform and improves the efficiency and speed of data processing. By automatically distributing the quality inspection success list from the submission pool to the target platform using the preset cycle, efficient resource utilization and time management can be achieved. For example, data transmission can be selected during non-peak hours to reduce the load on the target platform.

[0100] Step 306: Generate a quality inspection failure report based on all data quality inspection results in the current report to be inspected that fail to be inspected, and display the quality inspection failure report to the user so that the user can update the current report to be inspected based on the quality inspection failure report.

[0101] Among them, the quality inspection result is a success or failure. If the quality inspection result is a success, it is determined that the data has no abnormalities. If the quality inspection result is a failure, it is determined that the data has abnormalities. After determining the data quality inspection results of all the data in the current report to be tested, all data items with data quality inspection results of failed quality inspection are screened out, and a quality inspection failure report is generated based on the data content information of the data items that failed quality inspection, the reasons for the quality inspection failure, and the relevant data quality standards. The generated quality inspection failure report is displayed to the user, and the user can update the report to be tested through the detailed information provided in the quality inspection failure report. For example, according to the quality inspection failure report, correct erroneous data, supplement missing information, or further review the data. By promptly feeding back the data that failed quality inspection to the user, it is ensured that the problem data can be quickly identified and resolved, thereby improving the overall data quality.

[0102] Step 307: Receive the target return data sent by the target platform, and extract all data primary keys in the target return data.

[0103] Among them, the data primary key is used to identify different data, such as data serial number, etc. The target return data is the data obtained after the target platform organizes the quality inspection success detailed list after receiving it from the reporting data pool. Specifically, after the quality inspection success detailed list passes through the reporting data pool, data loss or data errors may occur. In order to avoid the situation where the target platform does not receive all correct and complete target data, the target platform summarizes and organizes the target data after receiving the target data. When idle, the target return data is generated according to the organized target data, and the target return data is sent to the data quality inspection platform. After receiving the target return data, the data quality inspection platform extracts all data primary keys of the target return data, so as to determine whether the target platform has received all correct and complete target data based on the data primary keys.

[0104] Step 308: Match each data primary key in the target return data with each data primary key in the quality inspection success detail table to obtain the matching results of each data primary key; if there is at least one primary key matching result that fails to match, generate updated data based on at least one primary key matching result and the quality inspection success detail table, and send the updated input data to the target platform.

[0105] Among them, the data primary key matching result is a successful match or a failed match. If the data primary key matching result of all data primary keys in the target return data is a successful match, it can be determined that the target platform has received all correct and complete target data. If there is at least one primary key matching result of a failed match, it means that the target data received by the target platform is inconsistent with the data in the successful quality inspection detail table generated by the data quality inspection platform. Furthermore, the data quality inspection platform can determine the data items corresponding to the primary keys that failed to match based on these failed primary keys and the successful quality inspection detail table, and determine them as updated data. Send the updated data to the target platform to ensure that the target platform receives the correct data. If there is no primary key matching result of a failed match, the operation is not performed. This process corrects errors that may occur during data transmission or processing, and ensures the integrity and accuracy of the data.

[0106] Step 309: Send at least one data primary key corresponding to a primary key matching failure result to the reporting data pool, so that the reporting data pool deletes data of at least one data primary key corresponding to a primary key matching failure result in the target data.

[0107] Specifically, the target data sent by the data quality inspection platform is stored in the reporting data pool, and the reporting data pool can regularly delete the target data that has been sent. In order to facilitate the management of the target data, the reporting data pool will not delete the target data immediately after sending the target data to the target platform, but will delete the target data after a period of time. If there is at least one primary key matching result of a matching failure, it means that there is data that has failed to be transmitted or has been transmitted incorrectly in the data reporting platform, and in the data reporting platform, these data may be marked as successfully transmitted. In order to ensure the consistency and accuracy of the data in each processing link, the data primary key corresponding to the primary key matching result of the matching failure is sent to the reporting data pool. After receiving the data primary key corresponding to the primary key matching result of the matching failure, the reporting data pool deletes the data of at least one data primary key corresponding to the primary key matching result of the matching failure in the target data.

[0108] By sending the primary key of data that failed to match to the reporting data pool, the data in the reporting data pool maintains high quality and consistency, providing a reliable foundation for subsequent data processing and analysis.

[0109] The technical solution of this embodiment receives at least one report to be detected sent by at least one data source. According to the data source, business type and predetermined dimensional attributes corresponding to each report to be detected, at least one dimensional attribute to which each report to be detected belongs is determined. For each dimensional attribute in at least one dimensional attribute, the sub-target rule template corresponding to the current dimensional attribute is determined according to the current dimensional attribute and the predetermined rule template library. For each report to be detected in at least one report to be detected, the target rule template of the current report to be detected is generated according to the sub-target rule templates corresponding to the dimensional attributes to which the current report to be detected belongs. The data quality inspection result of the current report to be detected is obtained according to the target rule template corresponding to the current report to be detected. A quality inspection success detailed list is generated according to the data whose quality inspection results of all data in the current report to be detected are successfully detected. The quality inspection success detailed list is determined as the target data, and the target data is sent to the predefined reporting data pool, so that the reporting data pool sends the quality inspection success detailed list to the target platform according to the preset period. A quality inspection failure report is generated according to the data whose quality inspection results of all data in the current report to be detected are failed to be detected, and the quality inspection failure report is displayed to the user, so that the user updates the current report to be detected based on the quality inspection failure report. Receive the target return data sent by the target platform, and extract all the data primary keys in the target return data. Match each data primary key in the target return data with each data primary key in the quality inspection success detail table to obtain the matching results of each data primary key; if there is at least one primary key matching result that fails to match, generate update data according to at least one primary key matching result and the inspection success detail table, and send the updated input data to the target platform. Send at least one primary key matching result that corresponds to the data primary key of the failed match to the reporting data pool, so that the reporting data pool deletes the data of the data primary key corresponding to the failed match in the target data. The technical solution of this embodiment can record the data quality inspection details, which is helpful to maintain data quality and modify abnormal data in time. The data can be batch processed and forwarded to the target platform through the reporting data pool, which reduces the data processing pressure of the target platform. By matching each data primary key in the target return data with each data primary key in the quality inspection success detail table, it is ensured that the target platform receives all correct and complete target data. Child labor promptly deleted erroneous data in the reported data pool, ensuring that the data in the reported data pool maintained high quality and consistency, providing a reliable foundation for subsequent data processing and analysis.

[0110] Figure 4 Schematic diagram of the structure of a data quality inspection device provided in an embodiment of the present invention, which is suitable for executing the data quality inspection method provided in an embodiment of the present invention. Figure 4 As shown, the device may specifically include:

[0111] The data receiving module 401 is used to receive at least one report to be detected sent by at least one data source;

[0112] The attribute determination module 402 is used to determine at least one dimension attribute to which each report to be detected belongs according to the data source, business type and predetermined dimension attributes corresponding to each report to be detected;

[0113] The template determination module 403 is used to determine, for each dimension attribute of the at least one dimension attribute, a sub-target rule template corresponding to the current dimension attribute according to the current dimension attribute and a predetermined rule template library;

[0114] The template generation module 404 is used to generate a target rule template for the current report to be detected according to the sub-target rule templates corresponding to the dimensional attributes of the current report to be detected for each report to be detected in the at least one report to be detected;

[0115] The result determination module 405 is used to obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested;

[0116] The data sending module 406 is used to determine the target data according to the current report to be detected and the data quality inspection result of the current report to be detected, and send the target data to the target platform.

[0117] Optionally, the template determination module 403 is specifically used to: for each report to be detected, according to the current data source of the current report to be detected, search the rule template library for a data source rule template sub-library corresponding to the current data source;

[0118] According to the current business type corresponding to the current report to be detected, searching the data source rule template sub-library for all business rule templates corresponding to the current business type;

[0119] Searching for a sub-target rule template corresponding to the current dimension attribute in all business rule templates corresponding to the current business type;

[0120] If the data source rule template sub-library cannot be found, or the business rule template cannot be found, or the sub-target rule template corresponding to the current dimension attribute cannot be found, a sub-target rule template corresponding to the current dimension attribute is generated according to a predetermined quality inspection rule library.

[0121] Optionally, the template determination module 403 is further used to: extract all data identifiers of data belonging to the current dimension attribute in the current report to be detected according to the data source of the current report to be detected;

[0122] Generate a candidate rule template for the current dimension attribute to which the current report to be inspected belongs according to the data identifier and the quality inspection rule library, and generate a rule display page according to the candidate rule template;

[0123] A sub-target rule template of the current dimension attribute to which the current report to be detected belongs is determined based on the rule display page.

[0124] Optionally, the template determination module 403 is further used to: for each data identifier, search the quality inspection rule library for at least one identification rule corresponding to the current data identifier;

[0125] A candidate rule template for the current dimension attribute is generated according to all data identifiers and at least one identification rule corresponding to all data identifiers.

[0126] Optionally, the template determination module 403 is further configured to: if a rule operation instruction sent by a user through the rule display page is received within a preset time period, modify the identification rule in the rule display page according to the rule operation instruction; the rule operation instruction includes adding a rule, deleting a rule, and changing a rule status;

[0127] Determine the sub-goal rule template of the current dimension attribute according to the modified rule display page;

[0128] If no rule operation instruction is received from the user through the rule display page within the preset time, or a confirmation instruction is received from the user through the rule display page, the candidate rule template of the current dimensional attribute is determined as the sub-target rule template of the current dimensional attribute.

[0129] Optionally, the template determination module 403 is further used to: when the rule operation instruction is the adding rule, determine all addable rule items corresponding to the data identifier according to the data identifier corresponding to the operation instruction;

[0130] The addable rule items are displayed to the user, so that the user can add rules to the data identifier corresponding to the operation instruction according to the addable rule items.

[0131] Optionally, the template determination module 403 is further used to: store the sub-target rule template corresponding to the current dimension attribute generated according to the quality inspection rule library in the rule template library.

[0132] Optionally, the result determination module 405 is specifically used to: determine the detection function of each rule attribute in the target rule template of the current report to be detected and the execution order of each detection function according to a preset rule attribute function library;

[0133] Generate a rule adapter for the current report to be detected according to each detection function and the execution order of each detection function;

[0134] The rule adapter performs data quality inspection on the current report to be inspected to obtain a data quality inspection result of the current report to be inspected.

[0135] Optionally, the data sending module 406 is specifically used to: generate a quality inspection success detailed table for the data that has been successfully inspected according to the quality inspection results of all data in the current report to be inspected;

[0136] The quality inspection success list is determined as the target data, and the target data is sent to a predefined reporting data pool, so that the reporting data pool sends the quality inspection success list to the target platform according to a preset period.

[0137] Optionally, the data sending module 406 is also used to: generate a quality inspection failure report based on all data quality inspection results in the current report to be tested that fail to detect, and display the quality inspection failure report to the user so that the user updates the current report to be tested based on the quality inspection failure report.

[0138] Optionally, the data sending module 406 is further used to: receive the target return data sent by the target platform, and extract all data primary keys in the target return data;

[0139] Matching each data primary key in the target returned data with each data primary key in the quality inspection success detail table to obtain each data primary key matching result;

[0140] If there is at least one primary key matching result that is a matching failure, update data is generated according to the at least one primary key matching result and the quality inspection success detail table, and the update data is sent to the target platform.

[0141] Optionally, the data sending module 406 is further used to: send the data primary key corresponding to the at least one primary key matching result being a matching failure to the reporting data pool, so that the reporting data pool deletes the data of the data primary key corresponding to the at least one primary key matching result being a matching failure in the target data.

[0142] The data quality inspection device provided in the embodiment of the present invention can execute the data quality inspection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, reference can be made to the description in any method embodiment of the present invention.

[0143] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, referring to Figure 5 , Figure 5 The electronic device 12 shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 5 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0144] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0145] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0146] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0147] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0148] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0149] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a data quality inspection method provided in an embodiment of the present invention: receiving at least one report to be inspected sent by at least one data source; determining at least one dimension attribute to which each report to be inspected belongs according to the data source, business type and predetermined dimension attributes corresponding to each report to be inspected; for each dimension attribute in the at least one dimension attribute, determining the sub-target rule template corresponding to the current dimension attribute according to the current dimension attribute and a predetermined rule template library; for each report to be inspected in the at least one report to be inspected, generating a target rule template for the current report to be inspected according to the sub-target rule templates corresponding to the dimension attributes to which the current report to be inspected belongs; obtaining the data quality inspection result of the current report to be inspected according to the target rule template corresponding to the current report to be inspected; determining the target data according to the current report to be inspected and the data quality inspection result of the current report to be inspected, and sending the target data to the target platform.

[0150] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a data quality inspection method provided by all the embodiments of the present invention is implemented: receiving at least one report to be inspected sent by at least one data source; determining at least one dimension attribute to which each report to be inspected belongs according to the data source, business type and predetermined dimension attributes corresponding to each report to be inspected; for each dimension attribute in the at least one dimension attribute, determining the sub-target rule template corresponding to the current dimension attribute according to the current dimension attribute and a predetermined rule template library; for each report to be inspected in the at least one report to be inspected, generating a target rule template for the current report to be inspected according to the sub-target rule templates corresponding to each dimension attribute to which the current report to be inspected belongs; obtaining the data quality inspection result of the current report to be inspected according to the target rule template corresponding to the current report to be inspected; determining the target data according to the data quality inspection result of the current report to be inspected and the current report to be inspected, and sending the target data to the target platform. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0152] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0153] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data quality inspection method provided in any embodiment of the present application.

[0155] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0156] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data quality inspection method, characterized in that: The method comprises: Receiving at least one report to be detected sent by at least one data source; Determine at least one dimensional attribute to which each report to be detected belongs according to a data source, a business type, and predetermined dimensional attributes corresponding to each report to be detected; For each dimensional attribute of the at least one dimensional attribute, according to the current dimensional attribute and a predetermined rule template library, determine a sub-target rule template corresponding to the current dimensional attribute; For each report to be detected in the at least one report to be detected, generating a target rule template for the current report to be detected according to the sub-target rule templates corresponding to each dimensional attribute to which the current report to be detected belongs; Obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested; The target data is determined according to the current report to be detected and the data quality inspection result of the current report to be detected, and the target data is sent to the target platform.

2. The method according to claim 1, characterized in that According to the current dimension attribute and a predetermined rule template library, a sub-goal rule template corresponding to the current dimension attribute is determined, including: For each report to be detected, searching the rule template library for a data source rule template sub-library corresponding to the current data source according to the current data source of the report to be detected; According to the current business type corresponding to the current report to be detected, searching the data source rule template sub-library for all business rule templates corresponding to the current business type; Searching for a sub-target rule template corresponding to the current dimension attribute in all business rule templates corresponding to the current business type; If the data source rule template sub-library cannot be found, or the business rule template cannot be found, or the sub-target rule template corresponding to the current dimension attribute cannot be found, a sub-target rule template corresponding to the current dimension attribute is generated according to a predetermined quality inspection rule library.

3. The method according to claim 2, characterized in that Generate a sub-goal rule template corresponding to the current dimension attribute according to a predetermined quality inspection rule library, including: Extract all data identifiers of data belonging to the current dimension attribute in the current report to be detected according to the data source of the current report to be detected; Generate a candidate rule template for the current dimension attribute to which the current report to be inspected belongs according to the data identifier and the quality inspection rule library, and generate a rule display page according to the candidate rule template; A sub-target rule template of the current dimension attribute to which the current report to be detected belongs is determined based on the rule display page.

4. The method according to claim 3, characterized in that: Generating a candidate rule template for the current dimension attribute to which the current report to be inspected belongs according to the data identifier and the quality inspection rule library includes: For each data identifier, searching the quality inspection rule library for at least one identifier rule corresponding to the current data identifier; A candidate rule template for the current dimension attribute is generated according to all data identifiers and at least one identification rule corresponding to all data identifiers.

5. The method according to claim 3, characterized in that: Determining a sub-target rule template of a current dimension attribute to which the current report to be detected belongs based on the rule display page includes: If a rule operation instruction sent by the user through the rule display page is received within a preset time period, modify the identification rule in the rule display page according to the rule operation instruction; the rule operation instruction includes adding a rule, deleting a rule, and changing a rule status; Determine the sub-goal rule template of the current dimension attribute according to the modified rule display page; If no rule operation instruction is received from the user through the rule display page within the preset time, or a confirmation instruction is received from the user through the rule display page, the candidate rule template of the current dimensional attribute is determined as the sub-target rule template of the current dimensional attribute.

6. The method according to claim 5, characterized in that Modifying the identification rule in the rule display page according to the rule operation instruction includes: When the rule operation instruction is the adding rule, all addable rule items corresponding to the data identifier are determined according to the data identifier corresponding to the operation instruction; The addable rule items are displayed to the user, so that the user can add rules to the data identifier corresponding to the operation instruction according to the addable rule items.

7. The method according to claim 2, characterized in that After generating the sub-target rule template corresponding to the current dimension attribute according to the predetermined quality inspection rule library, the method further includes: storing the sub-target rule template corresponding to the current dimension attribute generated according to the quality inspection rule library in the rule template library.

8. The method according to claim 1, characterized in that The data quality inspection results of the current report to be tested are obtained according to the target rule template corresponding to the current report to be tested, including: Determine, according to a preset rule attribute function library, the detection function of each rule attribute in the target rule template of the current report to be detected and the execution order of each detection function; Generate a rule adapter for the current report to be detected according to each detection function and the execution order of each detection function; The rule adapter performs data quality inspection on the current report to be inspected to obtain a data quality inspection result of the current report to be inspected.

9. The method according to claim 1, characterized in that: Determining target data according to the current report to be detected and the data quality inspection result of the current report to be detected, and sending the target data to the target platform, including: Generate a quality inspection success detailed table for the successfully inspected data according to the quality inspection results of all data in the current report to be inspected; The quality inspection success list is determined as the target data, and the target data is sent to a predefined reporting data pool, so that the reporting data pool sends the quality inspection success list to the target platform according to a preset period.

10. The method according to claim 9, characterized in that The method further comprises: A quality inspection failure report is generated for all data quality inspection results that fail the inspection in the current report to be inspected, and the quality inspection failure report is presented to the user, so that the user updates the current report to be inspected based on the quality inspection failure report.

11. The method according to claim 9, characterized in that The method comprises: Receive target return data sent by the target platform, and extract all data primary keys in the target return data; Matching each data primary key in the target returned data with each data primary key in the quality inspection success detail table to obtain each data primary key matching result; If there is at least one primary key matching result that is a matching failure, update data is generated according to the at least one primary key matching result and the quality inspection success detail table, and the update data is sent to the target platform.

12. The method according to claim 11, characterized in that The method also includes: sending the data primary key corresponding to the at least one primary key matching result being a matching failure to the reporting data pool, so that the reporting data pool deletes the data of the data primary key corresponding to the at least one primary key matching result being a matching failure in the target data.

13. A data quality inspection device, characterized in that: include: A data receiving module, used for receiving at least one report to be detected sent by at least one data source; An attribute determination module, used to determine at least one dimension attribute to which each report to be detected belongs according to a data source, a business type and predetermined dimension attributes corresponding to each report to be detected; A template determination module, configured to determine, for each dimensional attribute of the at least one dimensional attribute, a sub-target rule template corresponding to the current dimensional attribute according to the current dimensional attribute and a predetermined rule template library; A template generation module, used for generating a target rule template for the current report to be detected according to the sub-target rule templates corresponding to the dimensional attributes of the current report to be detected for each report to be detected in the at least one report to be detected; A result determination module is used to obtain the data quality inspection result of the current report to be tested according to the target rule template corresponding to the current report to be tested; The data sending module is used to determine the target data according to the current report to be detected and the data quality inspection result of the current report to be detected, and send the target data to the target platform.

14. A data quality inspection electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data quality inspection method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data quality inspection method as described in any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data quality inspection method according to any one of claims 1 to 12.

Images