Efficient data quality detection method

By defining data quality detection rules and configuring the basic rule database, using NiFi and XXL-JOB for data quality detection process orchestration and scheduling exploration, the problem that the existing technology cannot effectively detect and manage data quality, and achieve full-link data quality Standardized system and closed-loop control have improved data accuracy and consistency.

CN118916345APending Publication Date: 2024-11-08SHANGHAI INSPUR CLOUD COMPUTING SERVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411002553.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art cannot effectively detect and manage the completeness, accuracy, consistency and timeliness of large amounts of data, especially in the context of complex data sources, and lacks efficient data quality detection methods.

Method used

Provide an efficient data quality detection method, by defining data quality detection rules, configuring the basic rule database and business rule database, using NiFi and XXL-JOB for data quality detection process orchestration and scheduling exploration, supporting data exploration of multiple data sources and Improve quality, form a full-link data quality specification system and a closed-loop management and control system.

Benefits of technology

Achieve comprehensive detection and improvement of data quality, support resource allocation and scheduling, flexibly utilize NiFi data stream processing capabilities, and schedule various types of data sources through XXL-JOB to form a historical library and a problem library, improving data accuracy and consistency sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916345A_ABST
    Figure CN118916345A_ABST
Patent Text Reader

Abstract

The invention discloses an efficient data quality detection method, and relates to the technical field of big data. Comprising the steps of 1, defining a data quality detection rule, 2, carrying out data quality detection process arrangement according to the data quality detection rule based on NiFi, 3, starting a NiFi data quality detection process, and carrying out data exploration through XXL-JOB operation scheduling related data quality detection exploration tasks, and 4, according to a problem library formed by data problems found during data exploration, carrying out data quality detection on the data problems. And obtaining a historical library according to experience, and performing quality improvement and reporting on the data problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a method, which relates to the technical field of big data, and in particular to an efficient data quality detection method. Background Art

[0002] In the information age, the scale of data has increased significantly, and manual detection methods are no longer applicable. In particular, as data sources become more complex, data processing, management, and analysis require additional technical means. With the rise of technologies such as deep learning, cloud computing, parallel computing, and distributed systems, the requirements for data quality detection technology are getting higher and higher. However, there is currently no perfect method to detect the integrity, accuracy, consistency, and timeliness of large amounts of data. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides an efficient data quality detection method, which facilitates data quality improvement and data quality assessment, and creates a full-link data quality specification system and a closed-loop control system.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides an efficient data quality detection method, comprising:

[0006] Step 1: Define data quality detection rules:

[0007] The quality inspection dimensions are specified through data quality inspection rules.

[0008] Configure the basic rule base,

[0009] According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library;

[0010] Step 2: Orchestrate the data quality inspection process based on NiFi according to the data quality inspection rules: Configure the data quality inspection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standard is stored in the database;

[0011] Step 3: Start the NiFi data quality detection process, and run the relevant data quality detection exploration task through XXL-JOB to perform data exploration. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduling data exploration task according to the data exploration results;

[0012] Step 4: Based on the problem library formed by the data problems discovered during data exploration, the historical library is obtained based on experience, and the quality of data problems is improved and reported.

[0013] Furthermore, in step 1 of the efficient data quality detection method, the quality detection dimensions are specified through data quality detection rules, including: accuracy dimension, redundancy dimension, completeness dimension, correlation dimension, uniqueness dimension, standardization dimension, consistency dimension and timeliness dimension of quality detection.

[0014] Furthermore, in the step 1 of the described efficient data quality detection method, configuring the basic rule base includes: configuring SQL detection and repeatability detection of the basic rule base, defining SQL detection of multiple data sources, wherein the data sources include MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaMeng, KingBase, HighGo and Osca, defining related classes to implement SQL detection of data sources, and summarizing and counting the total amount of incoming SQL data and the total amount of problems.

[0015] Furthermore, in step 1 of the described efficient data quality detection method, basic rules are associated with the data source or data table to be detected through a business rule library, including: maintaining the business rules according to the association, wherein business rules are defined, and in the business rules, basic rules or corresponding SQL are configured for data tables and columns, the detection type is set to single table detection, MySQL data source and library, table and target value, the library and table for storing the data source are set, and the verification rules, parameter names, parameter descriptions, parameter types and parameter values ​​are recorded in the rule list.

[0016] Furthermore, in the data exploration in step 3 of the efficient data quality detection method, the data exploration step specifically includes:

[0017] Select specific data sources and data tables to configure the exploration information: Determine the exploration scope based on the data source and data table. The exploration scope includes the full amount and sampling. Set the sampling data volume. Select the scheduling mode. Automatically execute the detection task according to the time set by the scheduling mode. Explore all or part of the null value rate, repetition rate, data length, data value range, dictionary distribution, sample data and data format of all fields under the data table.

[0018] Create a probe task scheduling interface, create a probe scheduling task through the probe task scheduling interface, schedule data probe tasks: implement task scheduling through the XXL-JOB distributed task scheduling framework, and run data probe tasks through the probe task ProbeTaskHandler method. Routes to different data sources according to the data source type, and obtains table probe results during task operation: number of fields, total data volume; field probe results: column name, data element, null value rate, repetition rate, data length, sample data, and data format.

[0019] View the data profiling results and optimize the parameters for scheduling data profiling tasks based on the data profiling results.

[0020] The present invention also provides an efficient data quality detection device, including a rule management module, a process arrangement module, a scheduling detection module and a feedback module.

[0021] The rule management module defines data quality detection rules:

[0022] The quality inspection dimensions are specified through data quality inspection rules.

[0023] Configure the basic rule base,

[0024] According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library;

[0025] The process orchestration module orchestrates the data quality detection process based on NiFi according to the data quality detection rules: configure the data quality detection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standard is stored in the database;

[0026] The scheduling and exploration module starts the NiFi data quality detection process, and performs data exploration by running and scheduling related data quality detection and exploration tasks through XXL-JOB. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduling data exploration task according to the data exploration results;

[0027] The feedback module forms a problem library based on the data problems discovered during data exploration, obtains a historical library based on experience, and makes quality improvements on data problems and reports them.

[0028] Furthermore, the rule management module of the efficient data quality detection device specifies the quality detection dimensions through data quality detection rules, including: accuracy dimension, redundancy dimension, integrity dimension, correlation dimension, uniqueness dimension, standardization dimension, consistency dimension and timeliness dimension of quality detection.

[0029] Furthermore, the rule management module of the efficient data quality detection device configures the basic rule base, including: configuring SQL detection and repeatability detection of the basic rule base, defining SQL detection of multiple data sources, where the data sources include MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaMeng, KingBase, HighGo and Osca, defining related classes to implement data source SQL detection, and summarizing and counting the total amount of incoming SQL data and the total amount of problems.

[0030] Furthermore, the rule management module of the described efficient data quality detection device associates basic rules with the data source or data table to be detected through a business rule library, including: maintaining business rules according to the association, wherein business rules are defined, and in the business rules, basic rules or corresponding SQL are configured for data tables and columns, the detection type is set to single table detection, MySQL data source and library, table and target value, the library and table for storing the data source are set, and the verification rules, parameter names, parameter descriptions, parameter types and parameter values ​​are recorded in the rule list.

[0031] Furthermore, when the scheduling exploration module of the efficient data quality detection device performs data exploration, it specifically includes:

[0032] Select specific data sources and data tables to configure the exploration information: Determine the exploration scope based on the data source and data table. The exploration scope includes the full amount and sampling. Set the sampling data volume. Select the scheduling mode. Automatically execute the detection task according to the time set by the scheduling mode. Explore all or part of the null value rate, repetition rate, data length, data value range, dictionary distribution, sample data and data format of all fields under the data table.

[0033] Create a probe task scheduling interface, create a probe scheduling task through the probe task scheduling interface, schedule data probe tasks: implement task scheduling through the XXL-JOB distributed task scheduling framework, and run data probe tasks through the probe task ProbeTaskHandler method. Routes to different data sources according to the data source type, and obtains table probe results during task operation: number of fields, total data volume; field probe results: column name, data element, null value rate, repetition rate, data length, sample data, and data format.

[0034] View the data profiling results and optimize the parameters for scheduling data profiling tasks based on the data profiling results.

[0035] The benefits of the present invention are:

[0036] The present invention can support resource allocation and scheduling, and more flexibly utilize NiFi's powerful data flow processing capabilities such as data extraction, conversion, and loading to realize user configuration and management of data flows in a distributed environment. Through XXL-JOB, data exploration of various types of data sources such as MySQL, Hive, and GreenPlum is scheduled; at the same time, with the help of NiFi's automated process orchestration tasks, in-depth exploration of data quality is carried out to form a historical library and a problem library, creating a full-link data quality specification system and a closed-loop control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0038] As a lightweight distributed task scheduling platform, XXL-JOB has been widely used in the field of distributed task scheduling due to its lightweight, high performance and easy scalability. NiFi is a powerful, flexible and easy-to-use data processing and distribution system suitable for various data processing and analysis scenarios.

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0040] Example 1

[0041] The present invention provides an efficient data quality detection method, comprising:

[0042] Step 1: Define data quality detection rules:

[0043] The quality inspection dimensions are specified through data quality inspection rules.

[0044] Configure the basic rule base,

[0045] According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library.

[0046] In step 1, the quality detection dimensions can be further specified through data quality detection rules, including: accuracy dimension, redundancy dimension, completeness dimension, correlation dimension, uniqueness dimension, standardization dimension, consistency dimension and timeliness dimension of quality detection.

[0047] Configuring the basic rule base may include: configuring SQL detection and duplication detection of the basic rule base, defining SQL detection for multiple data sources, including MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaMeng, KingBase, HighGo, and Osca, defining related classes to implement SQL detection for data sources, and summarizing and counting the total amount of incoming SQL data and the total amount of problems.

[0048] Associating basic rules with the data source or data table to be detected through the business rule library may include: maintaining business rules according to the association, defining business rules, configuring basic rules or corresponding SQL for data tables and columns in the business rules, setting the detection type to single table detection, MySQL data source and library, table and target value, setting the library and table for storing data sources, and recording verification rules, parameter names, parameter descriptions, parameter types and parameter values ​​in the rule list. Verification rules may involve digital verification, ID number detection, date verification, etc.

[0049] Step 2: Orchestrate the data quality inspection process based on NiFi according to the data quality inspection rules: Configure the data quality inspection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standards is stored in the database.

[0050] The data quality detection process is configured, for example, first read the Hive data source information, select the Hive data source, table name, incremental column and other basic information, set the number of single data files (the number is 10,000 in this invention), enable the incremental alarm flag, the lower limit of the alarm data volume and other advanced properties.

[0051] Perform standard validation on the data according to the process, for example, based on the source table of the validation field, the field check_result for recording error information, and the field check_mark for recording whether there is an error. You can add additional information such as validation fields and validation rules, as well as pull rules in advanced properties, route validation error information to the validation error library, and split and route problem data to the problem data detail table.

[0052] Perform data standard conversion, for example, configure conversion fields and conversion rules according to the conversion field source table, such as removing special characters, removing invisible characters at the beginning and end, etc. You can add, edit, and delete conversion fields and conversion rule column information. Route standard conversion error information to the conversion error library, and route standard conversion correct information to the standard verification library.

[0053] The data after standard conversion is subjected to secondary standard verification, and the data that meets the standard is stored in the library. Among them, the information that meets the standard can be written into the historical library according to a certain merge strategy. The NiFiMergeContent processor is used in the present invention, which is used to merge multiple stream files into one stream file, and according to the specified merge strategy, for example, the Merge Strategy is selected as the Bin-Packing Algorithm, and the stream files are merged into the historical library to facilitate data tracing and problem correction; in the present invention, the standard verification information is routed to the standard library to facilitate data reporting and data quality improvement. For the specific process arrangement, please refer to the quality inspection task scenario.

[0054] Step 3: Start the NiFi data quality detection process, and run the relevant data quality detection exploration tasks through XXL-JOB to perform data exploration. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduled data exploration task according to the data exploration results.

[0055] The data exploration may include:

[0056] Select specific data sources and data tables to configure the exploration information: Determine the exploration scope based on the data source and data table. The exploration scope includes the full amount and sampling. Set the sampling data volume. Select the scheduling mode. Automatically execute the detection task according to the time set by the scheduling mode. Explore all or part of the null value rate, repetition rate, data length, data value range, dictionary distribution, sample data and data format of all fields under the data table.

[0057] Create a probe task scheduling interface, create a probe scheduling task through the probe task scheduling interface, schedule data probe tasks: implement task scheduling through the XXL-JOB distributed task scheduling framework, and run data probe tasks through the probe task ProbeTaskHandler method. Routes to different data sources according to the data source type, and obtains table probe results during task operation: number of fields, total data volume; field probe results: column name, data element, null value rate, repetition rate, data length, sample data, and data format.

[0058] View the data profiling results and optimize the parameters for scheduling data profiling tasks based on the data profiling results.

[0059] Step 4: Based on the problem database formed by the data problems found during data exploration, the historical database is obtained based on experience, and the quality of data problems is improved and reported. For example, by establishing unified quality indicators and unified data models, automatic detection of data quality conditions and automatic discovery of data quality problems can be achieved, thereby promoting data quality improvement and ensuring data accuracy, consistency, and integrity.

[0060] Example 2

[0061] The present invention also provides an efficient data quality detection device, including a rule management module, a process arrangement module, a scheduling detection module and a feedback module.

[0062] The rule management module defines data quality detection rules:

[0063] The quality inspection dimensions are specified through data quality inspection rules.

[0064] Configure the basic rule base,

[0065] According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library;

[0066] The process orchestration module orchestrates the data quality detection process based on NiFi according to the data quality detection rules: configure the data quality detection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standard is stored in the database;

[0067] The scheduling and exploration module starts the NiFi data quality detection process, and performs data exploration by running and scheduling related data quality detection and exploration tasks through XXL-JOB. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduling data exploration task according to the data exploration results;

[0068] The feedback module forms a problem library based on the data problems discovered during data exploration, obtains a historical library based on experience, and makes quality improvements on data problems and reports them.

[0069] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0070] Similarly, the device of the present invention can support resource allocation and scheduling, and more flexibly utilize NiFi's powerful data flow processing capabilities such as data extraction, conversion, and loading to realize user configuration and management of data flows in a distributed environment. Through XXL-JOB, data exploration of various types of data sources such as MySQL, Hive, and GreenPlum is scheduled; at the same time, with the help of NiFi automated process orchestration tasks, in-depth exploration of data quality is carried out to form a historical library and a problem library, creating a full-link data quality specification system and a closed-loop control system.

[0071] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0072] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. An efficient data quality detection method, characterized by include: Step 1: Define data quality detection rules: The quality inspection dimensions are specified through data quality inspection rules. Configure the basic rule base, According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library; Step 2: Orchestrate the data quality inspection process based on NiFi according to the data quality inspection rules: Configure the data quality inspection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standard is stored in the database; Step 3: Start the NiFi data quality detection process, and run the relevant data quality detection exploration task through XXL-JOB to perform data exploration. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduling data exploration task according to the data exploration results; Step 4: Based on the problem library formed by the data problems discovered during data exploration, the historical library is obtained based on experience, and the quality of data problems is improved and reported.

2. According to the method of claim 1, the method is characterized in that In step 1, the quality detection dimensions are specified through data quality detection rules, including: accuracy dimension, redundancy dimension, completeness dimension, relevance dimension, uniqueness dimension, standardization dimension, consistency dimension and timeliness dimension of quality detection.

3. The method for detecting high-efficiency data quality according to claim 1, characterized in that In step 1, configure the basic rule base, including: configure SQL detection and duplication detection of the basic rule base, define SQL detection for multiple data sources, including MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaMeng, KingBase, HighGo, and Osca, define related classes to implement SQL detection for data sources, and summarize and count the total amount of incoming SQL data and the total amount of problems.

4. The method for detecting high-efficiency data quality according to claim 1, characterized in that In step 1, the basic rules are associated with the data source or data table to be detected through the business rule library, including: maintaining the business rules according to the association, defining the business rules, configuring the basic rules or corresponding SQL for the data table and column in the business rules, setting the detection type to single table detection, MySQL data source and library, table and target value, setting the library and table for storing the data source, and recording the verification rules, parameter name, parameter description, parameter type and parameter value in the rule list.

5. The method for detecting high-efficiency data quality according to claim 1, characterized in that The data exploration in step 3 specifically includes: Select specific data sources and data tables to configure the exploration information: Determine the exploration scope based on the data source and data table. The exploration scope includes the full amount and sampling. Set the sampling data volume. Select the scheduling mode. Automatically execute the detection task according to the time set by the scheduling mode. Explore all or part of the null value rate, repetition rate, data length, data value range, dictionary distribution, sample data and data format of all fields under the data table. Create a probe task scheduling interface, create a probe scheduling task through the probe task scheduling interface, schedule data probe tasks: implement task scheduling through the XXL-JOB distributed task scheduling framework, and run data probe tasks through the probe task ProbeTaskHandler method. Routes to different data sources according to the data source type, and obtains table probe results during task operation: number of fields, total data volume; field probe results: column name, data element, null value rate, repetition rate, data length, sample data, and data format. View the data profiling results and optimize the parameters for scheduling data profiling tasks based on the data profiling results.

6. An efficient data quality detection device, characterized in that It includes rule management module, process orchestration module, scheduling exploration module and feedback module. The rule management module defines data quality detection rules: The quality inspection dimensions are specified through data quality inspection rules. Configure the basic rule base, According to the data quality detection rules, the basic rules are associated with the data source or data table to be detected through the business rule library; The process orchestration module orchestrates the data quality detection process based on NiFi according to the data quality detection rules: configure the data quality detection process, in which the data is verified according to the process, the data standard is converted, the data after the standard conversion is verified again, and the data that meets the standard is stored in the database; The scheduling and exploration module starts the NiFi data quality detection process, and performs data exploration by running and scheduling related data quality detection and exploration tasks through XXL-JOB. When exploring data, select the specific data source and data table to configure the exploration information, create an exploration task scheduling interface, create an exploration scheduling task through the exploration task scheduling interface, schedule the data exploration task, view the data exploration results, and optimize the parameters of the scheduling data exploration task according to the data exploration results; The feedback module forms a problem library based on the data problems discovered during data exploration, obtains a historical library based on experience, and makes quality improvements on data problems and reports them.

7. The high-efficiency data quality detection device according to claim 6, characterized in that The rule management module specifies the quality detection dimensions through data quality detection rules, including: accuracy dimension, redundancy dimension, integrity dimension, relevance dimension, uniqueness dimension, standardization dimension, consistency dimension and timeliness dimension of quality detection.

8. The high-efficiency data quality detection device according to claim 6, characterized in that The rule management module configures the basic rule base, including: configuring SQL detection and duplication detection of the basic rule base, defining SQL detection for multiple data sources, including MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaMeng, KingBase, HighGo, and Osca, defining related classes to implement SQL detection for data sources, and summarizing and counting the total amount of incoming SQL data and the total amount of problems.

9. The high-efficiency data quality detection device according to claim 6, characterized in that The rule management module associates the basic rules with the data source or data table to be detected through the business rule library, including: maintaining the business rules according to the association, defining the business rules, configuring the basic rules or corresponding SQL for the data table and column in the business rules, setting the detection type to single table detection, MySQL data source and library, table and target value, setting the library and table for storing the data source, and recording the verification rules, parameter name, parameter description, parameter type and parameter value in the rule list.

10. The high-efficiency data quality detection device according to claim 6, characterized in that When scheduling the exploration module to explore data, it specifically includes: Select specific data sources and data tables to configure the exploration information: Determine the exploration scope based on the data source and data table. The exploration scope includes the full amount and sampling. Set the sampling data volume. Select the scheduling mode. Automatically execute the detection task according to the time set by the scheduling mode. Explore all or part of the null value rate, repetition rate, data length, data value range, dictionary distribution, sample data and data format of all fields under the data table. Create a probe task scheduling interface, create a probe scheduling task through the probe task scheduling interface, schedule data probe tasks: implement task scheduling through the XXL-JOB distributed task scheduling framework, and run data probe tasks through the probe task ProbeTaskHandler method. Routes to different data sources according to the data source type, and obtains table probe results during task operation: number of fields, total data volume; field probe results: column name, data element, null value rate, repetition rate, data length, sample data, and data format. View the data profiling results and optimize the parameters for scheduling data profiling tasks based on the data profiling results.