Data quality inspection method and device, storage medium and computer program product

By configuring data accounts and rules for the data to be tested, and using the preset processing engine to automatically perform data quality inspection tasks, the problem of single rule dimensions and inability to meet multi-dimensional scenario-based quality inspection in the existing technology is solved, and different types of data are realized The comprehensive quality inspection has improved the efficiency and accuracy of quality inspection.

CN118964417APending Publication Date: 2024-11-15CHINA MERCHANTS BANK
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410929067.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing data quality inspection methods support a single rule dimension, which cannot meet the multi-dimensional scenario-based quality inspection needs, and cannot conduct comprehensive and complete quality inspections of different types of data.

Method used

By configuring data accounts and data quality inspection rules for the data to be inspected, using the preset processing engine to automatically perform data quality inspection tasks, supporting multi-dimensional scenario-based quality inspection settings, it can perform quality inspections for offline data and streaming real-time data. And add statistical analysis function of data.

Benefits of technology

It improves the efficiency and accuracy of data quality inspection, enriches and improves the observation dimensions of quality inspection results, and achieves comprehensive and perfect quality inspection of different types of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964417A_ABST
    Figure CN118964417A_ABST
Patent Text Reader

Abstract

The invention discloses a data quality inspection method and device, a storage medium and a computer program product, and relates to the technical field of data processing, and the method comprises the steps: obtaining to-be-inspected data; configuring a data account and a data quality inspection rule for the data to be subjected to quality inspection based on a preset configuration rule; configuring a data quality inspection task for the data to be subjected to quality inspection according to the data account and the data quality inspection rule; and distributing the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result. A multi-dimensional data quality inspection rule supports more-dimensional scene quality inspection setting, a processing engine with a statistical analysis function automatically executes a data quality inspection task and performs quality inspection on off-line data and streaming real-time data, quality inspection on different types of data in multi-dimensional scene quality inspection is realized, and a comprehensive and perfect quality inspection result is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data quality inspection method, device, storage medium and computer program product. Background Art

[0002] Data quality inspection is a key step to ensure data accuracy and reliability. With the continuous growth of enterprise data and the complexity of data application scenarios, the requirements for data quality are increasing. The current data quality inspection method is mainly based on executing the quality inspection tasks generated by the quality inspection rules corresponding to each column of data to be inspected, and obtaining the quality inspection results of each column of data to be inspected. However, the data quality inspection method supports a single rule dimension, incomplete data quality analysis, and a single data set type, which makes this method unable to meet the needs of modern enterprises.

[0003] Therefore, how to conduct quality inspection on different types of data in multi-dimensional scenario-based quality inspection and obtain comprehensive and complete quality inspection results is a problem that needs to be solved urgently.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data quality inspection method, device, storage medium and computer program product, aiming to solve how to perform quality inspection on different types of data in multi-dimensional scenario-based quality inspection and obtain comprehensive and complete quality inspection results.

[0006] To achieve the above objectives, the present application proposes a data quality inspection method, which includes:

[0007] Obtain the data to be inspected;

[0008] Configuring data accounts and data quality inspection rules for the data to be quality inspected based on preset configuration rules;

[0009] Configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule;

[0010] The data quality inspection task is distributed to a preset processing engine for processing to obtain a data quality inspection result.

[0011] In one embodiment, the step of configuring a data account and a data quality inspection rule for the data to be quality inspected based on a preset configuration rule includes:

[0012] Determine data quality characteristic requirements and data storage method based on the data to be inspected;

[0013] A configuration rule is determined based on the data quality characteristic requirement and the data storage method.

[0014] In one embodiment, the step of configuring a data account and a data quality inspection rule for the data to be quality inspected based on a preset configuration rule includes:

[0015] Configuring a data account for the data to be inspected according to preset configuration rules;

[0016] Designing data quality inspection rule types according to the data quality characteristic requirements and the data storage method, and storing the data quality inspection rule types to obtain a data quality inspection rule library;

[0017] Searching the data quality inspection rule library based on the preset configuration rule to obtain a data quality inspection rule search result;

[0018] According to the data quality inspection rule retrieval result, data quality inspection rules are configured for the data to be quality inspected.

[0019] In one embodiment, the step of configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule includes:

[0020] Determining whether the data account is valid;

[0021] If yes, configuring a data quality inspection task for the data to be inspected based on the data quality inspection rule;

[0022] If not, a valid data account is created, and a data quality inspection task is configured for the data to be quality inspected based on the data quality inspection rule.

[0023] In one embodiment, if yes, the step of configuring a data quality inspection task for the data to be inspected based on the data quality inspection rule includes:

[0024] Determining whether the data quality inspection rule is applicable to the data account;

[0025] If yes, then the data quality inspection rule is associated with the data account, quality inspection task information is confirmed, and a data quality inspection task is configured for the data to be quality inspected according to the quality inspection task information;

[0026] If not, a quality inspection rule applicable to the data account is created, and the quality inspection rule applicable to the data account is associated with the data account, and a data quality inspection task is configured for the data to be quality inspected.

[0027] In one embodiment, the step of automatically executing the data quality inspection task based on a preset processing engine to obtain the data quality inspection result includes:

[0028] Get a real-time data processing framework;

[0029] Establishing a parsing engine and a quality inspection statistics distribution engine according to the real-time data processing framework;

[0030] Establishing a quality inspection engine and a statistics engine based on the parsing engine and the quality inspection statistics distribution engine;

[0031] The parsing engine, the quality inspection statistics distribution engine, the quality inspection engine and the statistics engine are combined to obtain a processing engine.

[0032] In one embodiment, the step of distributing the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result includes:

[0033] Distributing the data quality inspection task to the parsing engine to obtain the data quality inspection task containing a unified data format;

[0034] Distributing the data quality inspection tasks containing the unified data format to the quality inspection engine and the statistics engine based on the quality inspection statistics distribution engine;

[0035] Based on the quality inspection engine and the statistical engine, the data quality inspection task containing the unified data format is processed to obtain the data quality inspection result.

[0036] In addition, to achieve the above objectives, the present application also proposes a data quality inspection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data quality inspection method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the data quality inspection method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data quality inspection method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] By configuring data accounts and data quality inspection rules for the data to be inspected, and by setting multi-dimensional data quality inspection rules, more dimensional scenario-based quality inspection settings can be supported; data quality inspection tasks are configured for the data to be inspected according to the data accounts and data quality inspection rules, and the data quality inspection tasks are automatically executed using the preset processing engine to obtain data quality inspection results. The processing engine automatically executes data quality inspection tasks, which not only improves the efficiency and accuracy of data quality inspection, but also can perform quality inspection on offline data and streaming real-time data, increase the statistical analysis function of the data, enrich and improve the observation dimension of the quality inspection results, and realize quality inspection of different types of data in multi-dimensional scenario-based quality inspection, and obtain comprehensive and complete quality inspection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the data quality inspection method of this application;

[0044] Figure 2 A flowchart of the second embodiment of the data quality inspection method of this application is provided;

[0045] Figure 3 An example diagram of data quality inspection rules provided for the data quality inspection method of this application;

[0046] Figure 4 This is a schematic diagram of the main process of configuring data quality inspection tasks in the data quality inspection method of this application;

[0047] Figure 5 A flowchart of the fifth embodiment of the data quality inspection method of this application is provided;

[0048] Figure 6 This is a schematic diagram of the Flink+Kafka streaming real-time data framework for the data quality inspection method of this application;

[0049] Figure 7 This is a schematic diagram of the specific construction ideas of the data quality inspection method for this application;

[0050] Figure 8 This is a schematic diagram of the module structure of the data quality inspection device according to an embodiment of the present application;

[0051] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the data quality inspection method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: configure data accounts and data quality inspection rules for the data to be quality inspected, configure data quality inspection tasks for the data to be quality inspected according to the data accounts and data quality inspection rules, and the preset processing engine automatically executes the data quality inspection tasks to obtain comprehensive and complete quality inspection results.

[0056] This application takes into account: obtaining the data set to be quality inspected and the set of quality inspection rules input by the user, and generating and executing the quality inspection tasks for each column of the data to be quality inspected according to the corresponding quality inspection rules. However, the quality inspection rule set has a single dimension and only supports single-column rule settings. It does not support cross-column scenarios and rule settings for the entire data set. The quality inspection task only supports data quality monitoring, not statistical analysis of the data. The data quality inspection results are incomplete, and during the execution of the quality inspection task, it only supports offline data quality inspection and analysis of batch jobs, but does not support quality inspection and analysis of streaming real-time data, and cannot perform quality inspection on different types of data.

[0057] Therefore, this application provides a solution to configure data accounts and data quality inspection rules for the data to be inspected according to the preset configuration rules, and to increase the support for scenario-based quality inspection settings in more dimensions through different types of data quality inspection rules. The preset processing engine is used to automatically execute data quality inspection tasks configured according to data accounts and data quality inspection rules, increase the statistical analysis function of the data, and enrich the observation dimension of data quality. The processing engine can perform quality inspection and analysis on streaming real-time data, thereby obtaining comprehensive and complete quality inspection results.

[0058] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data quality inspection tool, etc. The following takes the data quality inspection tool as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, the present application embodiment provides a data quality inspection method, referring to Figure 1, Figure 1 This is a flow chart of the first embodiment of the data quality inspection method of this application.

[0060] In this embodiment, the data quality inspection method includes steps S10 to S40:

[0061] Step S10, obtaining data to be inspected;

[0062] Data quality check is an important process to ensure that data meets the preset quality standards. Through data quality check, errors and omissions in the data can be discovered and repaired to ensure the accuracy and reliability of the data.

[0063] The data to be inspected is collected from data sources such as sensors, databases, and log files according to predetermined frequencies and conditions. Before the data is used for quality inspection, the collected raw data can be preprocessed, including steps such as cleaning the data and removing invalid and incomplete records. After the data to be inspected is collected, it is transmitted to the data quality inspection tool and stored in the database of the data quality inspection tool for subsequent data quality inspection and post-quality inspection inspection.

[0064] Step S20, configuring a data account and data quality inspection rules for the data to be quality inspected based on preset configuration rules;

[0065] A data account is a configuration item used to manage and define information related to a data source for data quality inspection. Data quality inspection rules are standards used to evaluate data quality, defining the conditions or attributes that data should meet.

[0066] Automatically parse data metadata by configuring data accounts. Automatically parse data metadata includes parsing data entity lists, data table structures, and data column information, providing visualized basic metadata information to facilitate data quality inspection rule association.

[0067] To configure a data account for the data to be inspected, you first need to determine the data type of the data account based on the characteristics of the data to be inspected and business requirements. The data types of the data account include database, data warehouse, file system, etc. According to the data type of the data account, enter the data source information of the data to be inspected. The data source information includes database connection information, data warehouse connection information, file system path, etc.

[0068] Furthermore, by parsing the data structure of the data to be inspected, the data structure information in the data account is configured, the data structure information includes a data table structure and data column information, and by inputting the data structure information into the data account, it is ensured that the data account contains all the data information of the data to be inspected.

[0069] Furthermore, Grant statements are used to assign permissions to data accounts. In order to reduce data security risks, certain permissions are assigned to data accounts to protect data from being illegally obtained or tampered with.

[0070] Finally, match the appropriate data quality inspection rules for the data to be inspected based on the data metadata in the data account. In the quality inspection rule configuration interface, choose to create a new quality inspection rule or select an existing rule. Depending on the tool or platform, fill in the basic information of the rule, such as the rule name, description, importance, etc. If you create a new quality inspection rule, you need to configure the specific content of the quality inspection rule, including the fields for quality inspection, define the conditions and standards for quality inspection, and set the strength of the rule. If you choose an existing rule, you need to make a selection based on the specific quality inspection goal.

[0071] Step S30, configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule;

[0072] The data metadata in the data account enables the rapid configuration of data quality inspection tasks, while the data quality inspection rules clarify the basic direction of data quality inspection. Data quality inspection tools usually provide a visual interface to facilitate the configuration of data quality inspection tasks. No code needs to be written, and data quality inspection tasks can be configured through simple operations.

[0073] Specifically, in a feasible implementation manner, the step S30 of configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule may include:

[0074] First, confirm the permissions of the data account to ensure that the account has all the permissions required to perform the quality inspection task. Since the data to be inspected contains a large amount of data, a large number of data accounts and corresponding data quality inspection rules will be generated during the process of configuring data accounts and data quality inspection rules for the data to be inspected. The permissions of the data accounts are inconsistent, and the defined data quality inspection rules are also inconsistent. Further confirm the permissions of the data accounts for the data to be inspected that need to be configured with data quality inspection tasks to ensure the accuracy of the subsequent data quality inspection results.

[0075] Furthermore, the data quality inspection rules are analyzed in detail to understand the data items that need to be checked, the inspection standards, possible error types, etc., and the complexity of the rules and the required processing resources are evaluated in order to allocate appropriate computing resources for the quality inspection tasks.

[0076] Furthermore, according to the analysis results of the data quality inspection rules and the data account permissions, the specific data quality inspection task information is configured. The analyzed data quality inspection rules are applied to the quality inspection tasks, and the configuration of the data quality inspection tasks is realized by importing the rules as queries, scripts or predefined templates into the task configuration.

[0077] In this implementation, the permissions of the data accounts required to perform the quality inspection tasks are confirmed in detail to ensure that the data quality inspection tasks can be performed under accounts with sufficient permissions, thereby avoiding the risk of task failure or data leakage due to insufficient permissions. Detailed analysis of data quality inspection rules helps to better understand quality inspection requirements and evaluate the complexity of the rules and the required processing resources, providing an important basis for subsequent task configuration.

[0078] The above is only a feasible implementation of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0079] Step S40: distribute the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result.

[0080] The preset processing engine is an engine combination that integrates various data quality inspection functions. Each engine usually contains a series of algorithms, rules, templates and tools to automatically perform data quality inspection tasks, realize quality inspection of different types of data in multi-dimensional scenario-based quality inspection, and obtain comprehensive and complete quality inspection results.

[0081] Data quality inspection results include data item status, exception or error information, data overview, processing suggestions, and other data quality inspection information presented in reports, logs, database tables or other forms.

[0082] The processing engine will verify the data to be inspected according to the data quality inspection rules and data quality inspection content recorded in the data quality inspection task, record the verification results of each data item, and store them. Based on the verification results, the processing engine can generate detailed data quality inspection results, which include an overall overview of the data, the number of data items that passed the verification, the number of data items that failed the verification, and specific information about anomalies or errors.

[0083] This embodiment provides a data quality inspection method, which supports scenario-based quality inspection settings in more dimensions by configuring data accounts and data quality inspection rules for the data to be inspected, and by setting multi-dimensional data quality inspection rules; configuring data quality inspection tasks for the data to be inspected according to the data accounts and data quality inspection rules, and automatically executing data quality inspection tasks using a preset processing engine to obtain data quality inspection results. The processing engine automatically executes data quality inspection tasks, which not only improves the efficiency and accuracy of data quality inspection, but also can perform quality inspection on offline data and streaming real-time data, increase the statistical analysis function of the data, enrich and improve the observation dimension of the quality inspection results, and realize quality inspection of different types of data in multi-dimensional scenario-based quality inspection, and obtain comprehensive and complete quality inspection results.

[0084] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0085] On this basis, please refer to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the data quality inspection method of this application.

[0086] In this embodiment, before step S20 of configuring a data account and data quality inspection rules for the data to be inspected based on preset configuration rules, steps S121 to S122 are also included:

[0087] Step S121, determining data quality characteristic requirements and data storage methods according to the data to be inspected;

[0088] Data quality characteristic requirements are important criteria for evaluating whether data meets usage requirements. Different data types and business needs have different requirements for data quality.

[0089] By clarifying the data quality characteristic requirements, we can ensure that the data to be inspected meets the preset standards, thereby improving the credibility and reliability of the data. Data quality characteristic requirements include accuracy requirements, consistency requirements, uniqueness requirements, timeliness requirements, completeness requirements and validity requirements.

[0090] Data storage methods refer to the form or technology of data storage and management in computer systems, which determines how data is organized, accessed, modified, and protected. Different data storage methods have different access efficiency and query performance.

[0091] Determine the data storage method by analyzing the type and structure of the data to be inspected, and comprehensively consider the data storage method of the data to be inspected in combination with business needs, data analysis goals, industry standards, and the specific use of the data.

[0092] Step S122: determining configuration rules based on the data quality characteristic requirements and the data storage method.

[0093] Configuration rules include the quality characteristic requirements of the data to be tested, data storage methods, data source information of the data to be tested, data metadata parsing rules, and a series of standards that can be used to configure data quality inspection rules and data accounts for the data to be tested.

[0094] Design the framework of configuration rules according to the data quality characteristics and data storage methods. The basic framework of configuration rules should include: rule types (such as verification rules, conversion rules, cleaning rules, etc.), rule triggering conditions, rule execution order, etc. After determining the framework of configuration rules, determine the management method of rules, which includes viewing, modifying, and deleting the rule list.

[0095] It should be noted that configuration rules are not the same as data quality inspection rules. Configuration rules are the standards for configuring data accounts and data quality inspection rules for data to be inspected. Through configuration rules, accurate matching between data to be inspected and data accounts and data quality inspection rules can be achieved. Data quality inspection rules are the standards for quality inspection of data.

[0096] In this embodiment, configuration rules are determined according to data quality characteristic requirements and the data storage method, so that the data to be quality inspected can be accurately matched with the data account and data quality inspection rules, providing a foundation for the subsequent construction of the basic capabilities of the data quality inspection tool.

[0097] Based on the first embodiment and / or the second embodiment of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can be referred to the above introduction, and will not be repeated in the following.

[0098] In this embodiment, the step S20 of configuring the data account and the data quality inspection rules for the data to be quality inspected based on the preset configuration rules includes steps S201 to S204:

[0099] Step S201, configuring a data account for the data to be inspected according to a preset configuration rule;

[0100] By configuring the data account, data metadata is automatically parsed. The analysis includes parsing the data entity list, data table structure and data column information, providing visual metadata basic information for data quality inspection task configuration, facilitating data quality inspection rule association and quickly implementing data quality inspection task configuration.

[0101] It should be noted that data accounts can provide data acquisition capabilities for the processing engine to perform data quality inspection. The processing engine is the core capability of the data quality inspection tool and can realize the automated execution of data quality inspection tasks. The data information that actually needs to be inspected in the quality inspection task is automatically acquired and inspected by the system through the data account definition and the relevant information in the data quality inspection task.

[0102] The data account includes data source information, data type information, data metadata parsing rules, etc., and the configuration rules specify the specific data account configuration process. First, determine the data source information in the data account according to the configuration rules. The data source information includes the database name, data table, data column name, and data acquisition method.

[0103] Further, the data type information and data meta information parsing rules in the data account are determined according to the configuration rules. The data type of the data to be inspected is determined by analyzing the data storage method in the configuration rules, and the data type of the data to be inspected can determine the parsing rules of the data meta information.

[0104] Step S202, designing data quality inspection rule types according to the data quality characteristic requirements and the data storage method, and storing the data quality inspection rule types to obtain a data quality inspection rule library;

[0105] According to the relevant characteristic requirements of data quality and combined with the data storage method, four types of rules are designed, including single-column rules, cross-column rules, cross-row rules and cross-table rules. Data quality rule types include more than a dozen types of rules such as not nullable types and grammatical constraint types, which overcome the limitation of only being able to set up single-column rule quality inspections and add support for scenario-based quality inspection settings in more dimensions.

[0106] On this basis, reference Figure 3 , Figure 3 An example diagram of data quality inspection rules provided for the data quality inspection method of this application. Data quality inspection rules are divided into four categories: single-column rules, cross-column rules, cross-row rules, and cross-table rules. The above four categories of rules are subdivided into multiple data quality inspection rules including non-nullable categories, grammatical constraint categories, business constraint categories, etc.

[0107] It is understandable that, in addition to the four types of rules mentioned above, different data quality inspection rules can also be determined according to specific business needs. All data quality inspection rules are stored in the data quality inspection rule library to ensure the integrity and security of the data quality inspection rules.

[0108] Step S203, searching the data quality inspection rule library based on the preset configuration rule to obtain a data quality inspection rule search result;

[0109] The data quality inspection rule library contains a large number of rules. By searching through preset configuration rules, you can quickly find rules that meet specific conditions or requirements, thereby improving the efficiency of data quality inspection.

[0110] To search in the data quality inspection rule base according to the configuration rules, you need to construct the corresponding search statement. By using logical operators to combine the configuration rules, using wildcards to fuzzy match keywords, executing the search statement in the data quality inspection rule base, obtaining a list of rules that meet the conditions, and obtaining the data quality inspection rule search results.

[0111] Step S204: configuring data quality inspection rules for the data to be quality inspected according to the data quality inspection rule retrieval result.

[0112] According to the search results retrieved from the data quality inspection rule database, appropriate data quality inspection rules are configured for the data to be quality inspected. In order to ensure the accuracy of subsequent quality inspection, the data quality inspection rules can be further refined.

[0113] Move data quality inspection rules forward to data production, warehousing and circulation links, monitor and inspect data quality including total amount and content in advance and give reminders, so as to improve data quality and achieve data quality goals of accuracy, consistency, uniqueness, timeliness, completeness and effectiveness, and give full play to the functions that data quality inspection tools should have.

[0114] In this embodiment, the configuration rules are determined according to the data quality characteristics and data storage methods, and accordingly, data accounts are configured for the data to be inspected, data quality inspection rule types are designed, a data quality inspection rule library is constructed, suitable rules are retrieved, and data quality inspection rules are configured for the data to be inspected. The rules designed according to the data quality characteristics can ensure that the data meets specific quality standards, thereby improving the accuracy and reliability of the data. The data quality inspection rule library solves the limitation of only being able to set single-column rule quality inspection, and adds support for scenario-based quality inspection settings in more dimensions.

[0115] Based on the above embodiments of the present application, a fourth embodiment of the method of the present application is proposed. In the fourth embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be repeated in the following.

[0116] In this embodiment, the step S30 of configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule includes steps S301 to S303:

[0117] Step S301, determining whether the data account is valid;

[0118] In order to protect the security of the data to be inspected, certain permissions are set for each data account during the configuration of the data account, and the permissions between data accounts are different. Judging whether a data account is valid is used to verify whether the data account exists, that is, whether the data account has the permission for data quality inspection.

[0119] Step S302: If yes, configuring a data quality inspection task for the data to be inspected based on the data quality inspection rule;

[0120] If the data account has data quality inspection permissions, obtain the metadata information contained in the data account, and provide a basis for configuring data quality inspection tasks based on data quality inspection rules. Visualized metadata information facilitates data quality inspection rule association and quickly implements data quality inspection task configuration.

[0121] Specifically, in a feasible implementation manner, if yes, then the step S302 of configuring the data quality inspection task for the to-be-inspected data based on the data quality inspection rule may include steps S1 to S3:

[0122] Step S1, determining whether the data quality inspection rule is applicable to the data account;

[0123] The purpose of data quality inspection rule configuration is to establish and maintain a data quality inspection rule base, providing multiple types of quality inspection rules, such as single-column rules, cross-column rules, cross-row rules, cross-table rules, etc., as well as rule types such as not nullable and grammatical constraints. Users can select different rules for configuration as needed to meet different data quality inspection requirements. The role of judging whether the rules are applicable to the data account link is to ensure the validity and applicability of the selected quality inspection rules, avoid configuration errors, and ensure that data quality inspection tasks can be accurately executed.

[0124] Step S2: if yes, then associate the data quality inspection rule with the data account, confirm the quality inspection task information, and configure the data quality inspection task for the data to be quality inspected according to the quality inspection task information;

[0125] If the data quality inspection rule is applicable to the data account and exists in the data quality inspection rule library, associate the data quality inspection rule with the data account, confirm the data quality inspection information, and complete the configuration of the data quality inspection task based on the data quality inspection information.

[0126] Data quality inspection tasks refer to the process of checking and analyzing data according to preset quality inspection rules to evaluate data quality, including task name, task description, execution method, and notification mechanism. For example, the execution method is to check the data one by one according to the data quality inspection rules.

[0127] Step S3: if not, create a quality inspection rule applicable to the data account, associate the quality inspection rule applicable to the data account with the data account, and configure a data quality inspection task for the data to be quality inspected.

[0128] If the data quality inspection rule is not applicable to the data account and does not exist in the data quality inspection rule library, you need to create a quality inspection rule applicable to the data account and associate the newly created data quality inspection rule with the data account to ensure that the data quality inspection task execution method can check the data one by one according to the data quality inspection rule.

[0129] In this embodiment, it is determined whether the data quality inspection rule is applicable to the data account, the data quality inspection rule is associated with the data account based on the judgment result, the quality inspection task information is confirmed, and the data quality inspection task is configured for the data to be quality inspected based on the quality inspection task information, to ensure that subsequent data quality inspection is carried out accurately and orderly.

[0130] The above is only a feasible implementation of step S302 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S302.

[0131] Step S303: if not, create a valid data account, and configure a data quality inspection task for the data to be inspected based on the data quality inspection rule.

[0132] If the data account does not have the data quality inspection authority, the data account needs to be rebuilt to enable the data quality inspection authority. After the new data account is built, a connectivity test is performed on the new data account. If the connectivity test passes, the data quality inspection task can be configured according to the content described in step S302.

[0133] It should be noted that if you encounter a data quality inspection task configuration for multiple data sources, you can first select data source 1 and configure the rules. After selecting the data source, continue to select data source 2 and configure the rules until all data sources are configured with data quality inspection rules. Finally, confirm the steps of the task information to complete the configuration of the data quality inspection task for multiple data sources.

[0134] In this embodiment, data quality inspection tasks are configured for the data to be inspected according to the data account and data quality inspection rules. The configuration of data accounts and data quality inspection rules can be automated without manual code writing, which greatly improves efficiency. The configuration is simple and fast through the visual interface, and quality inspection tasks can be quickly created and managed, gradually building the quality inspection capabilities of the data quality inspection tool, providing a basis for subsequent quality inspection of different types of data and obtaining comprehensive and complete quality inspection results.

[0135] In conjunction with the fourth embodiment of the present application, the following reference Figure 4 , Figure 4 This is a schematic diagram of the main process of configuring data quality inspection tasks. This section explains the main process of configuring data quality inspection tasks.

[0136] To configure a data quality inspection task, you must first select a data account. Based on the data account, determine whether the data account has the authority to inspect data and the specific data source information in the data account. If the data account has the authority, the account exists and you proceed to the next step. If the data account does not exist, you need to create a new data account. After the new data account passes the connection test, proceed to the next step.

[0137] Furthermore, associate the data account with the data quality inspection rule. If the data quality inspection rule is applicable to the data account, the rule exists, and the data quality inspection rule is associated with the data account, and the final step is to configure the quality inspection task. If the data quality inspection rule is not applicable to the data account, a new quality inspection rule needs to be created, and the data quality inspection rule is associated with the data account, and the final step is to configure the quality inspection task.

[0138] Finally, configure the quality inspection task based on the association results. Determine the quality inspection task information based on the data account and data quality inspection rules. After determining the quality inspection task information, complete the configuration of the quality inspection task based on the quality inspection task information.

[0139] Based on the above embodiments of the present application, a fifth embodiment of the present application is proposed. In the fifth embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later.

[0140] On this basis, please refer to Figure 5 , Figure 5 This is a flowchart of the fifth embodiment of the data quality inspection method of the present application.

[0141] In this embodiment, the step S341 to S344 are included before the step S40 of distributing the data quality inspection task to a preset processing engine for processing and obtaining the data quality inspection result:

[0142] Step S341, obtaining a real-time data processing framework;

[0143] The basic idea of ​​the data quality inspection method proposed in this application is to solve actual data quality problems, establish a data quality inspection framework, and gradually improve the data quality inspection capabilities. Configuring data accounts, data quality inspection rules, and data quality inspection tasks for the data to be inspected is to improve the quality inspection capabilities of the data quality inspection framework.

[0144] Obtain the Flink+Kafka streaming real-time data framework, and gradually build a data quality inspection framework based on the framework. Configure data accounts, data quality inspection rules, and data quality inspection tasks for the data to be quality inspected to improve the quality inspection capabilities of the data quality inspection framework. Finally, quality inspection of different types of data can be performed in multi-dimensional scenario-based quality inspection to obtain comprehensive and complete quality inspection results.

[0145] like Figure 6 As shown, Figure 6 This is a schematic diagram of the Flink+Kafka streaming real-time data framework for the data quality inspection method of this application. The Flink+Kafka streaming real-time data framework is a data framework obtained by performing logical architecture abstraction processing on the overall processing architecture based on the Kappa real-time data processing architecture.

[0146] The logical architecture abstraction of the overall processing architecture is actually to abstract data quality inspection into four parts, including data analysis, data quality inspection and statistics, result storage, and operational analysis. The Kappa real-time data processing architecture is a new architecture for streaming data processing, which consists of data source, data processing, storage, and operational components. The Kappa real-time data processing architecture can monitor data inflow in real time, verify it according to quality inspection rules, promptly identify data quality problems and notify relevant personnel; perform statistical analysis on data, such as calculating indicators such as total data volume, average, variance, etc., to fully understand data quality; perform batch quality inspection on historical data, analyze data quality trends, and provide a basis for data governance.

[0147] According to the Kappa real-time data processing architecture, the Flink+Kafka streaming real-time data framework is obtained by abstracting the logical architecture of the overall processing architecture. The processing link of the entire real-time data architecture is divided into four parts: parsing and sending, quality inspection and statistical distribution, engine processing, and storage operation. It is adapted to the parsing of source data with different structures and decoupled from the quality inspection and statistical logic of the data to meet the requirements of high cohesion and low coupling, thereby achieving the purpose of easy scalability.

[0148] Step S342, establishing a parsing engine and a quality inspection statistics distribution engine according to the real-time data processing framework;

[0149] First, according to the real-time data processing framework, select the appropriate parsing tool as the core component of the parsing engine, configure the appropriate parsing process, and combine the parsing tool and the parsing process to obtain a parsing engine that can normalize data with different structures.

[0150] Furthermore, a unified interface specification is defined in the parsing engine to facilitate interaction with different quality inspection engines and statistical engines. A configuration interface or interface is provided to allow users to define data distribution rules, such as distribution based on data type, keywords, data volume and other conditions.

[0151] Furthermore, the data processing requirements of different data to be inspected are obtained, and the event processing mechanism in the real-time data processing framework is used to implement the construction of the quality inspection statistics distribution engine according to the different data processing requirements.

[0152] Furthermore, after building the parsing engine and quality inspection statistics distribution engine, the parsing engine and quality inspection statistics distribution engine are tested and optimized to ensure that the parsing engine and quality inspection statistics distribution engine can be seamlessly integrated with other components (such as data sources, data storage, visualization tools, etc.) to form a complete data quality inspection link.

[0153] Step S343, establishing a quality inspection engine and a statistics engine based on the parsing engine and the quality inspection statistics distribution engine;

[0154] The parsing engine normalizes data of different structures into a unified data format, and the quality inspection and statistics distribution engine distributes data to different quality inspection engines and statistics engines based on different data processing requirements. The information interface of the quality inspection engine and statistics engine is constructed according to the standardized information interface defined in the parsing engine and the quality inspection and statistics distribution engine, and the processing modules of the quality inspection engine and statistics engine are determined according to the processed data format to ensure that data can be exchanged smoothly.

[0155] The quality inspection engine includes a real-time data quality inspection engine and a batch data quality inspection engine, which are deployed on the big data real-time platform and the container platform respectively. The real-time data quality inspection engine is mainly used to automatically monitor the total amount of business data entering the warehouse from multiple sources, and publish the quality inspection comparison results regularly according to the data quality inspection rules; the batch quality inspection engine performs content data quality inspection tasks on data tables or files through triggering and scheduled scheduling.

[0156] The statistical engine performs statistical analysis on the quality inspection data and builds the statistical engine by selecting or designing suitable statistical indicator algorithms. Since the statistical engine needs to process a large amount of data, distributed computing is used to build the performance optimization module of the statistical engine, and the plug-in architecture is used to build the scalability design module of the statistical engine.

[0157] Step S344, combining the parsing engine, the quality inspection statistics distribution engine, the quality inspection engine and the statistics engine to obtain a processing engine.

[0158] Combining the parsing engine, quality inspection and statistics distribution engine, quality inspection engine and statistics engine into a complete processing engine requires ensuring that these components can work together and connect seamlessly in the data flow.

[0159] First, in order to ensure smooth data exchange between different engines, a standardized data interface needs to be defined, including data format, transmission protocol, error handling mechanism, etc., so that each engine can understand each other's data and instructions.

[0160] Furthermore, the parsing engine, quality inspection engine, quality inspection statistics distribution engine and statistics engine are integrated. Each engine can be deployed and managed as an independent service or component using a microservice architecture or pipeline flow mode. To ensure that the communication between engines is reliable and efficient, data transmission can be carried out using message queues, API interfaces or data flow pipelines.

[0161] Furthermore, data flow control logic is implemented to ensure that data flows between engines according to predetermined paths and rules. The processing engine is tested and verified to ensure that it can process data in the expected manner and meet business requirements.

[0162] In this embodiment, a real-time data processing framework is obtained; a parsing engine and a quality inspection statistics distribution engine are established according to the real-time data processing framework, and a quality inspection engine and a statistics engine are further established according to the parsing engine and the quality inspection statistics distribution engine; the parsing engine, the quality inspection statistics distribution engine, the quality inspection engine and the statistics engine are combined to obtain a processing engine, so that the processing engine not only supports quality inspection of batch offline data and streaming real-time data, but also supports statistical analysis of the data, thereby providing more complete data quality inspection results.

[0163] Based on the above embodiments of the present application, a sixth embodiment of the method of the present application is proposed. In the sixth embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be repeated in the following.

[0164] In this embodiment, the step S40 of distributing the data quality inspection task to a preset processing engine for processing to obtain the data quality inspection result includes steps S401 to S403:

[0165] Step S401, distributing the data quality inspection task to the parsing engine to obtain the data quality inspection task containing a unified data format;

[0166] By defining a set of common data structures in the parsing engine, the data to be inspected contained in the data quality inspection task can be converted from data of different data formats and structures into data of a unified data format after being parsed by the parsing engine, thereby eliminating inconsistencies and ambiguities in the data, improving the quality and accuracy of the data, and accelerating the process of data quality inspection.

[0167] After the data to be inspected is distributed to the parsing engine, the parsing engine parses the data source of the data to be inspected according to the defined general data structure. During the parsing process, field mapping and format conversion are performed on the data source of each data to be inspected, and the data of the data to be inspected is converted into a unified format.

[0168] Step S402, distributing the data quality inspection task containing the unified data format to the quality inspection engine and the statistics engine based on the quality inspection statistics distribution engine;

[0169] The quality inspection and statistics distribution engine distributes the parsed data in a unified format to different quality inspection engines and statistics engines according to different data processing requirements. The quality inspection and statistics distribution engine enables flexible configuration of data quality inspection and statistics. Different quality inspection engines and statistics engines can be selected according to different scenarios and requirements, improving the scalability and maintainability of the system, so that each engine can be developed and optimized independently without affecting each other.

[0170] The quality inspection statistics distribution engine implements rule parsing and matching logic according to the data distribution rules defined by the user, and distributes the data to the corresponding quality inspection engine or statistics engine. In addition, the quality inspection statistics distribution engine can dynamically adjust the data distribution strategy according to the load of the engine to avoid single point overload.

[0171] Step S403: Processing the data quality inspection task including the unified data format based on the quality inspection engine and the statistical engine to obtain a data quality inspection result.

[0172] The quality inspection engine performs quality inspection on data according to the data quality inspection rules recorded in the quality inspection task. It checks whether a single data column meets the preset conditions according to the single-column rule. The preset conditions include non-empty, data type, value range, etc.; it checks the logical relationship between multiple data columns according to the cross-column rule; it checks the logical relationship between data rows according to the cross-row rule; and finally, it checks the logical relationship between different data tables according to the cross-table rule.

[0173] After the quality inspection engine performs quality inspection on the quality inspection data, a preliminary quality inspection result is obtained. The final quality inspection result needs to be combined with the result of the statistical engine analyzing the quality inspection data. Therefore, it is necessary to use the statistical engine to further analyze the quality inspection data.

[0174] The statistical engine analyzes the quality inspection data from three aspects: data distribution, data trend and data quality indicators. Data distribution is used to analyze the distribution of statistical data, such as the maximum, minimum, average, standard deviation, etc. of the data to be inspected; data trend is used to analyze the changing trend of data over time; data quality indicators are used to calculate data quality indicators, which include data completeness rate, data accuracy rate, etc.

[0175] By combining the preliminary quality inspection results of the quality inspection engine and the statistical analysis results of the quality inspection data by the statistical engine, a comprehensive and complete data quality inspection result is obtained, which solves the limitation of only being able to perform data quality inspection, increases the statistical analysis function of the data, and enriches the observation dimension of data quality.

[0176] In this embodiment, the parsing engine is used to unify the data format of the data to be inspected contained in the data quality inspection task to improve the accuracy of quality inspection. Based on the quality inspection statistics distribution engine, the data quality inspection task containing the unified data format is distributed to the quality inspection engine and the statistical engine; the data quality inspection task containing the unified data format is processed by the quality inspection engine and the statistical engine to obtain the data quality inspection result. The quality inspection engine not only supports data quality inspection and analysis of batch offline data, but also supports data quality inspection and analysis of streaming real-time data. The statistical engine adds the statistical analysis function of the data and enriches the observation dimension of data quality. The data quality inspection rules and data accounts contained in the data quality inspection task support more dimensional scenario-based quality inspection settings, so as to realize quality inspection of different types of data in multi-dimensional scenario-based quality inspection and obtain comprehensive and complete quality inspection results.

[0177] Reference Figure 7 As shown, Figure 7 This is the idea diagram for building the data quality inspection method for this application. The following is combined with Figure 7 , the specific construction ideas of the data quality inspection method of this application are explained.

[0178] The data quality inspection method of this application is mainly driven by solving actual data quality problems, establishing a data quality inspection framework, and gradually improving data quality inspection capabilities. The configuration of data quality inspection rules, data accounts, and data quality inspection tasks is to gradually improve the quality inspection capabilities of the preset processing engine.

[0179] Data quality inspection rules are designed according to data quality characteristics requirements, and data quality characteristics requirements are as follows: Figure 7 The data quality objectives shown are consistent, including accuracy, consistency, uniqueness, timeliness, completeness, and effectiveness. By moving the data quality verification rules forward to the data production, storage, and circulation links, the data quality including the total amount and content are monitored and checked in advance, and reminders are given to improve the data quality and achieve the data quality objectives of data accuracy, consistency, uniqueness, timeliness, completeness, and effectiveness. The data quality inspection tool should play its due role.

[0180] The core capabilities of data quality inspection tools include the associated configuration of data quality inspection objects and data quality inspection rules, data quality inspection task management, data quality inspection engine construction, data quality inspection task execution and quality inspection result release, notification mechanism, etc.

[0181] The two basic elements of data quality are data and rules for verifying whether the data meets the quality standards. Visual configuration management of data sources and data rules is the basic capability for establishing metadata information and data rule libraries for data quality inspection and building data quality inspection tools.

[0182] By analyzing the complete path of data application and combining it with the data quality issues that arise during the data application process, we can clarify the links and scope of data quality inspection tools for data quality inspection and realize the advancement of data quality verification rules.

[0183] Therefore, the overall architecture of the tool mainly abstracts the data source configuration and data quality inspection rule configuration, and on this basis builds quality inspection task configuration including content quality inspection tasks and multi-data total quality inspection tasks, builds real-time quality inspection engines and real-time statistics engines, and builds visual and notifiable data quality inspection operations, ultimately realizing general data quality inspection capabilities that can be quickly configured for various systems.

[0184] This application also provides a data quality inspection device, please refer to Figure 8 , the data quality inspection device comprises:

[0185] The data acquisition module 10 is used to acquire the data to be inspected;

[0186] The data to be inspected is collected from data sources such as sensors, databases, and log files according to predetermined frequencies and conditions. Before the data is used for quality inspection, the collected raw data can be preprocessed, including steps such as cleaning the data and removing invalid and incomplete records. After the data to be inspected is collected, it is transmitted to the data quality inspection tool and stored in the database of the data quality inspection tool for subsequent data quality inspection and post-quality inspection inspection.

[0187] Configuration module 20, used to configure data accounts and data quality inspection rules for the data to be quality inspected according to preset configuration rules;

[0188] By configuring data accounts, data metadata can be automatically parsed, including parsing data entity lists, data table structures, and data column information, providing visual metadata basic information to facilitate data quality inspection rule association.

[0189] A quality inspection task configuration module 30, configured to configure a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule;

[0190] Detailed confirmation of permissions for data accounts required to perform quality inspection tasks ensures that data quality inspection tasks can be performed under accounts with sufficient permissions, thereby avoiding the risk of task failure or data leakage due to insufficient permissions. Detailed analysis of data quality inspection rules helps to better understand quality inspection requirements, evaluate the complexity of rules and required processing resources, and provide important basis for subsequent task configuration.

[0191] The data quality inspection result module 40 is used to distribute the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result.

[0192] The data to be inspected is inspected according to the data quality inspection rules and data quality inspection content recorded in the data quality inspection task, and the verification results of each data item are recorded and stored. Based on the verification results, the processing engine can generate detailed data quality inspection results, which include an overall overview of the data, the number of data items that have passed the verification, the number of data items that have not passed the verification, and specific information about anomalies or errors.

[0193] The data quality inspection device provided by the present application adopts the data quality inspection method in the above embodiment to solve the technical problem of data quality inspection. Compared with the prior art, the beneficial effects of the data quality inspection device provided by the present application are the same as the beneficial effects of the data quality inspection method provided by the above embodiment, and the other technical features in the data quality inspection device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0194] The present application provides a data quality inspection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data quality inspection method in the above-mentioned embodiment one.

[0195] Reference below Fig. 9 , which shows a schematic diagram of the structure of a data quality inspection device suitable for implementing the embodiment of the present application. The data quality inspection device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The data quality inspection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0196] like Fig. 9 As shown, the data quality inspection device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data quality inspection device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data quality inspection device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data quality inspection device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0197] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0198] The data quality inspection device provided by the present application adopts the data quality inspection method in the above embodiment to solve the technical problem of data quality inspection. Compared with the prior art, the beneficial effects of the data quality inspection device provided by the present application are the same as the beneficial effects of the data quality inspection method provided by the above embodiment, and the other technical features in the data quality inspection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0199] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0200] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0201] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data quality inspection method in the above-mentioned embodiment.

[0202] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0203] The computer-readable storage medium may be included in the data quality inspection device; or may exist independently without being assembled into the data quality inspection device.

[0204] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data quality inspection device, the data quality inspection device: obtains the data to be inspected; configures the data account and data quality inspection rules for the data to be inspected based on preset configuration rules; configures the data quality inspection task for the data to be inspected according to the data account and the data quality inspection rules; automatically executes the data quality inspection task based on a preset processing engine to obtain the data quality inspection result.

[0205] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0206] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0207] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0208] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data quality inspection method, and can solve the technical problems of data quality inspection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data quality inspection method provided in the above-mentioned embodiment, and will not be repeated here.

[0209] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data quality inspection method when executed by a processor.

[0210] The computer program product provided in this application can solve the technical problem of data quality inspection. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data quality inspection method provided in the above embodiment, which will not be repeated here.

[0211] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data quality inspection method, characterized in that: The method comprises: Obtain the data to be inspected; Configuring data accounts and data quality inspection rules for the data to be quality inspected based on preset configuration rules; Configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule; The data quality inspection task is distributed to a preset processing engine for processing to obtain a data quality inspection result.

2. The method according to claim 1, characterized in that Before the step of configuring a data account and data quality inspection rules for the data to be quality inspected based on preset configuration rules, the following steps are also included: Determine data quality characteristic requirements and data storage method based on the data to be inspected; A configuration rule is determined based on the data quality characteristic requirement and the data storage method.

3. The method according to claim 2, characterized in that The step of configuring a data account and data quality inspection rules for the data to be quality inspected based on preset configuration rules comprises: Configuring a data account for the data to be inspected according to preset configuration rules; Designing data quality inspection rule types according to the data quality characteristic requirements and the data storage method, and storing the data quality inspection rule types to obtain a data quality inspection rule library; Searching the data quality inspection rule library based on the preset configuration rule to obtain a data quality inspection rule search result; According to the data quality inspection rule retrieval result, data quality inspection rules are configured for the data to be quality inspected.

4. The method according to claim 1, characterized in that The step of configuring a data quality inspection task for the data to be inspected according to the data account and the data quality inspection rule comprises: Determining whether the data account is valid; If yes, configuring a data quality inspection task for the data to be inspected based on the data quality inspection rule; If not, a valid data account is created, and a data quality inspection task is configured for the data to be quality inspected based on the data quality inspection rule.

5. The method according to claim 4, characterized in that If so, the step of configuring a data quality inspection task for the data to be inspected based on the data quality inspection rule includes: Determining whether the data quality inspection rule is applicable to the data account; If yes, then the data quality inspection rule is associated with the data account, quality inspection task information is confirmed, and a data quality inspection task is configured for the data to be quality inspected according to the quality inspection task information; If not, a quality inspection rule applicable to the data account is created, and the quality inspection rule applicable to the data account is associated with the data account, and a data quality inspection task is configured for the data to be quality inspected.

6. The method according to claim 1, characterized in that The step of distributing the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result includes: Get a real-time data processing framework; Establishing a parsing engine and a quality inspection statistics distribution engine according to the real-time data processing framework; Establishing a quality inspection engine and a statistics engine based on the parsing engine and the quality inspection statistics distribution engine; The parsing engine, the quality inspection statistics distribution engine, the quality inspection engine and the statistics engine are combined to obtain a processing engine.

7. The method according to claim 6, characterized in that The step of distributing the data quality inspection task to a preset processing engine for processing to obtain a data quality inspection result comprises: Distributing the data quality inspection task to the parsing engine to obtain the data quality inspection task containing a unified data format; Distributing the data quality inspection tasks containing the unified data format to the quality inspection engine and the statistics engine based on the quality inspection statistics distribution engine; Based on the quality inspection engine and the statistical engine, the data quality inspection task containing the unified data format is processed to obtain the data quality inspection result.

8. A data quality inspection device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data quality inspection method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data quality inspection method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data quality inspection method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Data quality detection method and device

    CN111913942A

  • Multi-source heterogeneous data quality verification method and system based on unified SQL

    CN113760681A

  • Real-time data quality detection method and system based on Flink

    CN113901035A

  • Banking business data quality detection method and device, storage medium and equipment

    CN117235133A

  • Data quality inspection method based on Spark engine

    CN117520434A