Quality detection method based on big data governance platform

Abstract

According to the quality detection method based on the big data governance platform, quality detection is divided into quality detection based on data standards and quality detection based on quality rules, and standardization and flexibility are considered at the same time; the data quality detection task and the data quality detection task are uniformly associated to related metadata to form an executable quality detection task, in quality detection, whether the data meet data standard requirements or not can be detected, various data problems can be detected according to service requirements, all requirements of data quality detection can be covered in an omnibearing mode, and the data quality detection efficiency is improved. A quality detection method with strong universality and good applicability is really established; different weights can be given to six types of quality detection including integrity, consistency, effectiveness, accuracy, uniqueness and timeliness according to the opinions of field experts, so that the data quality score is displayed more reasonably and accurately.

Description

Technical Field

[0001] The present invention relates to the field of quality inspection, and in particular to a quality inspection method based on a big data governance platform. Background Art

[0002] With the rapid development of digitalization and intelligence, the amount of data has exploded. The data governance platform maintains the integrity, availability, security and compliance of data through processes such as data collection, storage, processing, analysis and governance. It is an important infrastructure for digital transformation and intelligent construction. Data quality detection, as an important component module of the data governance platform, plays an increasingly important role.

[0003] However, quality testing based on data standards is usually done by manually writing testing rules, and automated testing is not possible; quality testing based on data problems is usually limited to relatively simple tests such as null values ​​and uniqueness, and is not very universal; only quality reports are generated, without data quality scores and scoring basis, making it impossible to understand the existing data quality status at a glance, and it is also impossible to quantify the subsequent data quality improvement effects.

[0004] Therefore, it is necessary to provide a quality inspection method based on the big data governance platform to solve the above technical problems. Summary of the invention

[0005] The purpose of the present invention is to provide a quality detection method based on a big data governance platform to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above purpose, the present invention provides a quality detection method based on a big data governance platform, comprising the following steps:

[0007] S1: Import national, industry, company and other standard documents;

[0008] S2: According to the requirements of the standard document, convert it into data standards in the form of naming specifications, data elements, code tables, etc., and implement the standard;

[0009] S3: Set common data quality templates by writing SQL;

[0010] S4: perform metadata collection;

[0011] S5: Add data standards and data quality templates to the corresponding metadata;

[0012] S6: According to different metadata, the parameters in the data quality template are instantiated to form specific executable quality detection rules;

[0013] S7: select metadata that needs quality inspection, form a quality inspection task and execute it;

[0014] S8: Configure data quality weights through expert method to display reasonable data quality scores.

[0015] Preferably, the naming specification is the formulation of the library table naming standard, including specification name, specification content, specification description and standard file mapping.

[0016] Preferably, the data element is the formulation of field standards, including identification class attributes, definition class attributes, relationship class attributes, representation class attributes, management class attributes, fusion class attributes and additional class attributes, among which the representation class attributes stipulate the data type, format, value range, and confidentiality level, and can reference standard documents and code tables.

[0017] Preferably, the code table is a formulation of a data dictionary table, including a code table code, a code table name, a standard mapping, and a code configuration.

[0018] Preferably, the data quality template setting in S3 includes the following steps:

[0019] S31: Fill in the template name and select the template type, where the template types include six categories: completeness, consistency, validity, accuracy, uniqueness and timeliness;

[0020] S32: Based on the purpose of quality inspection, it is implemented by using SQL. Because the template needs to be universal and needs to be applied to different database tables and fields later, the parts involving specific database tables and fields are replaced by formal parameters.

[0021] S33: Set alarm rules to issue an alarm prompt when the quality detection exceeds certain thresholds or ranges.

[0022] Preferably, when binding the data standard and data quality template with metadata, a certain collected metadata is selected, and the configured naming convention and data quality template can be added to the table metadata, and the data element can be added to the field metadata, thereby completing the binding of the data standard and data quality template with metadata.

[0023] Preferably, the part of the data quality template involving specific library tables and fields is replaced by formal parameters, and the formal parameters need to be manually instantiated according to different bound metadata to form specific executable quality detection rules.

[0024] Preferably, the quality inspection task is formed by filling in the task name, task description, and selecting the metadata that needs to be quality inspected. The quality inspection task can be executed once or periodically by configuring the execution cycle.

[0025] Preferably, the expert method for configuring data quality weights should assign different weights to the six major quality inspection categories of completeness, consistency, validity, accuracy, uniqueness and timeliness based on the opinions of domain experts, so as to display the data quality score more reasonably and accurately.

[0026] Compared with related technologies, the quality detection method based on the big data governance platform provided by the present invention has the following beneficial effects:

[0027] The present invention provides a quality detection method based on a big data governance platform, which decomposes quality detection into quality detection based on data standards and quality detection based on quality rules. The quality detection based on data standards can implement data standards according to standard regulations to generate executable data detection standards, which are characterized by being standardized and fixed; while quality rules are flexible and can be flexibly configured according to business needs. They can detect data quality problems, including incomplete, inconsistent, repeated, and incorrect data. The two are uniformly bound to the corresponding metadata to form an executable quality detection task. In this way, in one quality detection, it is possible to detect whether the data meets the data standard requirements, and various data problems can be detected according to business needs. All requirements for data quality detection can be fully covered, and a quality detection method with strong versatility and good applicability can be truly established; in addition, different weights can be given to six major quality detection categories, such as integrity, consistency, effectiveness, accuracy, uniqueness, and timeliness, according to the opinions of experts in the field, so as to more reasonably and accurately display data quality scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a quality inspection method based on a big data governance platform provided by the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0030] Please refer to Figure 1 ,in, Figure 1 A flow chart of a quality detection method based on a big data governance platform provided by the present invention. A quality detection method based on a big data governance platform comprises the following steps:

[0031] S1: Import national, industry, company and other standard documents;

[0032] S2: According to the requirements of the standard document, convert it into data standards in the form of naming specifications, data elements, code tables, etc., and implement the standard;

[0033] S3: Set common data quality templates by writing SQL;

[0034] S4: perform metadata collection;

[0035] S5: Add data standards and data quality templates to the corresponding metadata;

[0036] S6: According to different metadata, the parameters in the data quality template are instantiated to form specific executable quality detection rules;

[0037] S7: select metadata that needs quality inspection, form a quality inspection task and execute it;

[0038] S8: Configure data quality weights through expert method to display reasonable data quality scores.

[0039] The naming specification is the formulation of the naming standard for library tables, including specification name, specification content, specification description and standard file mapping.

[0040] The data element is the formulation of field standards, including identification class attributes, definition class attributes, relationship class attributes, representation class attributes, management class attributes, fusion class attributes and additional class attributes. Among them, the representation class attributes stipulate the data type, format, value range, and confidentiality level, and can reference standard documents and code tables.

[0041] The code table is the formulation of a data dictionary table, including code table coding, code table name, standard mapping, and code configuration.

[0042] The data quality template setting in S3 includes the following steps:

[0043] S31: Fill in the template name and select the template type, where the template types include six categories: completeness, consistency, validity, accuracy, uniqueness and timeliness;

[0044] S32: Based on the purpose of quality inspection, it is implemented by using SQL. Because the template needs to be universal and needs to be applied to different database tables and fields later, the parts involving specific database tables and fields are replaced by formal parameters.

[0045] S33: Set alarm rules to issue an alarm prompt when the quality detection exceeds certain thresholds or ranges.

[0046] When binding the data standard and data quality template with metadata, select a certain collected metadata, add the configured naming convention and data quality template to the table metadata, and add the data element to the field metadata, thereby completing the binding of the data standard and data quality template with metadata.

[0047] The part of the data quality template involving specific library tables and fields is replaced by formal parameters. Here, the formal parameters need to be manually instantiated according to different bound metadata to form specific executable quality detection rules.

[0048] The quality inspection task is formed by filling in the task name, task description, and selecting the metadata that needs to be quality inspected. The quality inspection task can be executed once or periodically by configuring the execution cycle.

[0049] The expert method for configuring data quality weights should assign different weights to the six major quality tests of completeness, consistency, validity, accuracy, uniqueness and timeliness based on the opinions of domain experts, so as to display the data quality score more reasonably and accurately.

[0050] The working principle of a quality detection method based on a big data governance platform provided by the present invention is as follows:

[0051] When working, first import the national, industry, company and other standard files; download the national standards, industry standards, company standards and other standard files required for the project to the local computer, and then upload these standard files locally to the system.

[0052] According to the requirements of the standard document, it is converted into data standards in the form of naming specifications, data elements, code tables, etc., and the standards are implemented; the naming specifications are the formulation of the naming standards for the library tables, including the specification name, specification content, specification description, and standard file mapping, which can be set according to the requirements of the standard document. Data elements are the formulation of field standards, including identification attributes, definition attributes, relationship attributes, representation attributes, management attributes, fusion attributes, and additional attributes. Among them, the representation attributes stipulate the data type, format, value range, confidentiality level, etc., which can be set according to the requirements of the standard document. The code table is the formulation of the data dictionary table, including the code table code, code table name, standard mapping, and code configuration, which can be set according to the requirements of the standard document.

[0053] It should be noted that the parts of the standard that often require quality inspection, such as naming conventions, data types, data formats, value ranges, etc., can directly solidify the inspection procedures into the standard. When configuring quality inspection, there is no need to write SQL one by one according to the standard content. You only need to bind the standard to the data element, and when executing, the quality inspection of the data element based on the data standard can be automatically inspected.

[0054] Set up common data quality templates by writing SQL;

[0055] Data quality templates are used to implement quality detection based on quality rules. They can be flexibly configured according to business needs to supplement quality requirements that are not covered by standards. The setting of data quality templates is divided into the following three steps:

[0056] (1) Fill in the template name and select the template type. The template types include completeness, consistency, validity, accuracy, uniqueness, and timeliness;

[0057] (2) Based on the purpose of quality inspection, SQL is used to write and implement it. Because the template needs to be universal and needs to be applied to different database tables and fields later, the parts involving specific database tables and fields are replaced by formal parameters;

[0058] (3) Set alarm rules to issue an alarm when quality detection exceeds certain thresholds or ranges.

[0059] Conduct metadata collection; select different databases and tables and conduct one-time or periodic metadata collection on them.

[0060] Add data standards and data quality templates to the corresponding metadata; select a certain collected metadata, add the configured naming conventions and data quality templates to the table metadata, and add data elements to the field metadata, thereby completing the binding of data standards and data quality templates to metadata.

[0061] According to different metadata, the parameters in the data quality template are instantiated to form specific executable quality detection rules. Since the data quality template involves specific database tables and fields, the parameters need to be manually instantiated according to the different bound metadata to form specific executable quality detection rules.

[0062] Select the metadata that needs quality inspection, form a quality inspection task and execute it; fill in the task name, task description, and select the metadata that needs quality inspection to form a quality inspection task. The quality inspection task can be executed once or periodically by configuring the execution cycle.

[0063] Data quality weights are configured through the expert method to display reasonable data quality scores. According to the opinions of domain experts, different weights are assigned to quality tests in six dimensions, including completeness, consistency, validity, accuracy, uniqueness, and timeliness. All database tables that have undergone quality tests will be scored based on this weight. The specific rules are as follows:

[0064] Score of a rule = number of passed tests / total number of tests

[0065] The score of a quality dimension = the sum of the scores of the rules involved in the scoring of this dimension / the number of rules involved in the scoring of this dimension

[0066] Total quality score = (∑score of each quality dimension involved in the evaluation * expert weight) / 6

[0067] Compared with related technologies, the quality detection method based on the big data governance platform provided by the present invention has the following beneficial effects:

[0068] The present invention provides a quality detection method based on a big data governance platform, which decomposes quality detection into quality detection based on data standards and quality detection based on quality rules. The quality detection based on data standards can implement data standards according to standard regulations to generate executable data detection standards, which are characterized by being standardized and fixed; while quality rules are flexible and can be flexibly configured according to business needs. They can detect data quality problems, including incomplete, inconsistent, repeated, and incorrect data. The two are uniformly bound to the corresponding metadata to form an executable quality detection task. In this way, in one quality detection, it is possible to detect whether the data meets the data standard requirements, and various data problems can be detected according to business needs. All requirements for data quality detection can be fully covered, and a quality detection method with strong versatility and good applicability can be truly established; in addition, different weights can be given to six major quality detection categories, such as integrity, consistency, effectiveness, accuracy, uniqueness, and timeliness, according to the opinions of experts in the field, so as to more reasonably and accurately display data quality scores.

[0069] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A quality detection method based on a big data governance platform, characterized in that: The following steps are involved: S1: Import national, industry, company and other standard documents; S2: According to the requirements of the standard document, convert it into data standards in the form of naming specifications, data elements, code tables, etc., and implement the standard; S3: Set common data quality templates by writing SQL; S4: perform metadata collection; S5: Add data standards and data quality templates to the corresponding metadata; S6: According to different metadata, the parameters in the data quality template are instantiated to form specific executable quality detection rules; S7: select metadata that needs quality inspection, form a quality inspection task and execute it; S8: Configure data quality weights through expert method to display reasonable data quality scores.

2. According to the quality detection method based on the big data governance platform of claim 1, it is characterized in that: The naming specification is the formulation of the naming standard for library tables, including specification name, specification content, specification description and standard file mapping.

3. According to a quality detection method based on a big data governance platform according to claim 1, it is characterized in that: The data element is the formulation of field standards, including identification class attributes, definition class attributes, relationship class attributes, representation class attributes, management class attributes, fusion class attributes and additional class attributes. Among them, the representation class attributes stipulate the data type, format, value range, and confidentiality level, and can reference standard documents and code tables.

4. According to a quality detection method based on a big data governance platform according to claim 1, it is characterized in that: The code table is the formulation of a data dictionary table, including code table coding, code table name, standard mapping, and code configuration.

5. According to a quality detection method based on a big data governance platform according to claim 1, it is characterized in that: The data quality template setting in S3 includes the following steps: S31: Fill in the template name and select the template type, where the template types include six categories: completeness, consistency, validity, accuracy, uniqueness and timeliness; S32: Based on the purpose of quality inspection, it is implemented by using SQL. Because the template needs to be universal and needs to be applied to different database tables and fields later, the parts involving specific database tables and fields are replaced by formal parameters. S33: Set alarm rules to issue an alarm prompt when the quality detection exceeds certain thresholds or ranges.

6. A quality detection method based on a big data governance platform according to claim 1, characterized in that: When executing the quality inspection task, there should be two execution modes: one-time execution and periodic execution.

7. A quality detection method based on a big data governance platform according to claim 1, characterized in that: When binding the data standard and data quality template with metadata, select a certain collected metadata, add the configured naming convention and data quality template to the table metadata, and add the data element to the field metadata, thereby completing the binding of the data standard and data quality template with metadata.

8. The quality detection method based on the big data governance platform according to claim 1 is characterized in that: The part of the data quality template involving specific library tables and fields is replaced by formal parameters. Here, the formal parameters need to be manually instantiated according to different bound metadata to form specific executable quality detection rules.

9. The quality detection method based on the big data governance platform according to claim 1 is characterized in that: The quality inspection task is formed by filling in the task name, task description, and selecting the metadata that needs to be quality inspected. The quality inspection task can be executed once or periodically by configuring the execution cycle.

10. A quality detection method based on a big data governance platform according to claim 1, characterized in that: The expert method for configuring data quality weights should assign different weights to the six major quality tests of completeness, consistency, validity, accuracy, uniqueness and timeliness based on the opinions of domain experts, so as to display the data quality score more reasonably and accurately.

Images