A mass data quality report generation method based on an aggregation model
A data quality and quality reporting technology, applied in the field of data governance, can solve problems such as inability to support aggregation models, lack of definition of general model verification rules, and inability to form data quality verification schemes for massive data
Inactive Publication Date: 2019-04-23
福建南威软件有限公司
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] In the existing technical solutions, there are already some management methods for verifying data quality and generating data quality reports, but most of these technical solutions have the disadvantage of being unable to flexibly configure verification rules and cannot support mass data for verification
[0005] Patent application publication number [CN 108595563 A] cannot support the aggregation model based on offline calculation, and can only perform data quality verification analysis on conventional scale data
[0006] Patent application publication number [CN 107818106 A] does not define a general model and configurable verification rules, but only verifies data consistency, and cannot form a complete data quality verification scheme for massive data
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0021] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0022] It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and / or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and / or combina...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a mass data quality report generation method based on an aggregation model. The physical actual data is firstly converted into a row-column aggregation model through a definedrow aggregation model, a defined column aggregation model and the like, and the aggregation model is original integral data and can be split and combined according to aggregation characteristics, sothat the offline calculation and the parallel verification of a plurality of verification processing units can be supported. In addition, the result data for generating the large-scale data quality report is not a whole, but the data quality report results outputted by a plurality of data quality check processing units are uniformly digested and aggregated through a message queue, and finally, a data quality check analysis report capable of customizing check rules for large-scale structured data is generated. According to the method, the customization of a general verification rule can be supported, and the offline data verification of the mass large-scale data can be supported.
Description
technical field [0001] The invention relates to the field of data governance, in particular to a method for generating mass data quality reports based on an aggregation model. Background technique [0002] With the development of information technology, data has gradually become the most important resource of enterprise value, and the subsequent data quality problems have become more and more serious. Data quality problems such as data errors, absences, and inconsistencies are problems that enterprises must face , correct and valid data is the premise of data storage and analysis. [0003] With the development of Internet technology and various storage technologies, the scale of data stored by enterprises continues to grow, and the verification of massive and large-scale data has become an inevitable problem in enterprise data governance. [0004] There are already some management methods for verifying data quality and generating data quality reports in existing technical s...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/27G06F9/54
CPCG06F9/546G06F2209/548
Inventor 肖俊鑫
Owner 福建南威软件有限公司