A data quality governance method based on programmable components

Abstract

The present invention relates to a data quality governance method based on programmable components, wherein data acquisition components, data evaluation components, data modification components, data analysis components, process improvement components, data storage components, and data destruction components are programmable according to the data quality governance scope and quality governance objectives formulated by the data definition component. The beneficial effects of the present invention are as follows: the various parts used for data quality management are split into loose components, and when performing data quality management, the required components are programmable and used according to the data quality governance scope and quality governance objectives formulated in the data governance definition component, so that the components can cooperate with each other to avoid process waste and time waste, and improve data quality governance efficiency; data can be stored in a dispersed manner, improve data extraction efficiency, and avoid data extraction performance bottlenecks when processing massive data.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a data quality management method based on programmable components. Background Art

[0002] At present, in the field of data quality optimization, the industry mainstream uses centralized data quality management systems. Traditional centralized data quality management systems have achieved standardized management of verification rules, scheduling of rule execution time, and unified management of data quality reports, which has improved the efficiency and management level of data quality verification.

[0003] The limitation of this centralized data quality management system is that the quality management process is fixed and cannot be freely arranged according to different data quality management scopes and quality management goals, resulting in waste of process and data quality management time; using traditional database centralized storage, performance bottlenecks are prone to occur when processing massive data. Summary of the invention

[0004] The purpose of the present invention is to overcome the above problems existing in the prior art and provide a data quality management method based on programmable components.

[0005] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0006] A data quality governance method based on programmable components, including:

[0007] Data governance definition components used to establish data quality governance scope and quality governance objectives,

[0008] A data acquisition component for acquiring data,

[0009] Data evaluation components used to evaluate data,

[0010] A data modification component used to modify abnormal data.

[0011] Data analysis components for analyzing data,

[0012] Process improvement components for improving data quality governance processes,

[0013] Data storage components for decentralized storage of data,

[0014] Data destruction components for destroying data;

[0015] The data acquisition component, data evaluation component, data modification component, data analysis component, process improvement component, data storage component, and data destruction component are arranged according to the data quality governance scope and quality governance objectives formulated by the data definition component.

[0016] The data acquisition component includes a plurality of data acquisition sub-components for respectively collecting data according to different data sources.

[0017] Among them, the data evaluation component includes a uniqueness evaluation sub-component, an integrity evaluation sub-component, an accuracy evaluation sub-component, a consistency evaluation sub-component, a relevance evaluation sub-component, and a timeliness evaluation sub-component.

[0018] The data modification component includes a cross-validation method data correction subcomponent for correcting erroneous data and missing data, and a similarity comparison method data removal subcomponent for removing redundant data.

[0019] The data analysis component includes a regression analysis subcomponent, a factor analysis subcomponent, a fishbone diagram analysis subcomponent, a Pareto analysis subcomponent, and a matrix data analysis subcomponent.

[0020] Wherein, the process improvement component includes a process feedback sub-component and a process reengineering sub-component.

[0021] Wherein, the data storage component includes a plurality of independent data storage sub-components.

[0022] The beneficial effects of the present invention are: the various parts used for data quality management are split into loose components. When performing data quality management, the required components are arranged and used according to the data quality governance scope and quality governance goals formulated in the data governance definition component, so that the components can cooperate with each other to avoid process waste and time waste and improve data quality governance efficiency; data can be stored in a decentralized manner to improve data extraction efficiency and avoid data extraction performance bottlenecks when processing massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 It is a framework diagram of the data quality governance method in the present invention. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0026] like Figure 1 As shown, a data quality governance method based on programmable components includes:

[0027] Data governance definition components used to establish data quality governance scope and quality governance objectives,

[0028] A data acquisition component for acquiring data,

[0029] Data evaluation components used to evaluate data,

[0030] A data modification component used to modify abnormal data.

[0031] Data analysis components for analyzing data,

[0032] Process improvement components for improving data quality governance processes,

[0033] Data storage components for decentralized storage of data,

[0034] Data destruction components for destroying data;

[0035] Data acquisition components, data evaluation components, data modification components, data analysis components, process improvement components, data storage components, and data destruction components are orchestrated according to the data quality governance scope and quality governance objectives established by the data definition component.

[0036] The data acquisition component includes multiple data acquisition sub-components that collect data according to different data sources.

[0037] The data evaluation components include uniqueness evaluation sub-component, integrity evaluation sub-component, accuracy evaluation sub-component, consistency evaluation sub-component, relevance evaluation sub-component, and timeliness evaluation sub-component.

[0038] The data modification component includes a cross-validation method data correction subcomponent for correcting erroneous data and missing data, and a similarity comparison method data removal subcomponent for removing redundant data.

[0039] The data analysis component includes a regression analysis subcomponent, a factor analysis subcomponent, a fishbone diagram analysis subcomponent, a Pareto analysis subcomponent, and a matrix data analysis subcomponent.

[0040] The process improvement component includes the process feedback sub-component and the process reengineering sub-component.

[0041] The data storage component includes several independent data storage sub-components.

[0042] In the first embodiment, the data quality management scope is the kilowatts of electricity used by residents, and the quality management target is the integrity assessment of the data. The data governance definition component arranges and uses the data acquisition subcomponent and the integrity assessment subcomponent. The data acquisition subcomponent obtains the kilowatts of electricity used by residents to form a kilowatts database of electricity used by residents. The integrity assessment subcomponent performs integrity assessment on the data in the kilowatts database of electricity used by residents, and forms a data integrity assessment report after the assessment.

[0043] In the second embodiment, the data quality management scope is a certain transformer power transmission area, and the quality management target is power consumption analysis. The data governance definition component arranges and uses the data acquisition sub-component, data evaluation component, data modification component, and data analysis component. The data acquisition sub-component obtains the power consumption data in a certain transformer power transmission area to form a regional power consumption database. The data evaluation component is then used to evaluate the data in the regional power consumption database, and the abnormal data is modified by the data modification component. The data analysis component performs data analysis on the modified regional power consumption database and forms an analysis report.

[0044] In the third embodiment, the scope of data quality governance is a municipal power grid dispatching area, and the quality governance goal is to optimize the power grid dispatching quality. The data governance definition component arranges and uses the data acquisition component, data evaluation component, data modification component, data analysis component, process improvement component, data storage component, and data destruction component. The data acquisition component obtains the power grid dispatching data in a municipal power grid dispatching area to form the power grid dispatching database of the city. Then the data evaluation component evaluates the six properties of the data in the power grid dispatching database, on the one hand evaluating the six properties of the data and on the other hand finding abnormal data. The data modification component modifies the abnormal data in the power grid dispatching database. The data analysis component analyzes the modified power grid dispatching database to form an analysis report. The process improvement component gives feedback and / or changes the arrangement order, coordination mode, and processing results of the previous components. The data storage component saves the data that needs to be saved, and the data that does not need to be saved is completely destroyed by the data destruction component to prevent it from being stolen or misused.

[0045] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A data quality management method based on programmable components, characterized in that: include: Data governance definition components used to establish data quality governance scope and quality governance objectives, A data acquisition component for acquiring data, Data evaluation components used to evaluate data, A data modification component used to modify abnormal data. Data analysis components for analyzing data, A process improvement component for improving the data quality governance process, wherein the process improvement component includes a process feedback subcomponent and a process reengineering subcomponent. Data storage components for decentralized storage of data, Data destruction components for destroying data; The data acquisition component, data evaluation component, data modification component, data analysis component, process improvement component, data storage component, and data destruction component are arranged according to the data quality governance scope and quality governance objectives formulated by the data definition component; The scope of data quality governance is a municipal power grid dispatching area, and the quality governance goal is to optimize the power grid dispatching quality. The data governance definition component arranges and uses the data acquisition component, data evaluation component, data modification component, data analysis component, process improvement component, data storage component, and data destruction component. The data acquisition component acquires the power grid dispatching data in the municipal power grid dispatching area to form the city's power grid dispatching database. Then the data evaluation component evaluates the data in the power grid dispatching database to find out abnormal data. The data modification component modifies the abnormal data in the power grid dispatching database. The data analysis component performs data analysis on the modified power grid dispatching database to form an analysis report. The process improvement component gives feedback and / or changes the arrangement order, coordination method, and processing results of the data acquisition component, data evaluation component, data modification component, and data analysis component. The data storage component saves the data that needs to be saved, and the data that does not need to be saved is completely destroyed by the data destruction component to prevent it from being stolen or misused.

2. The data quality management method based on programmable components according to claim 1 is characterized in that: The data acquisition component includes a plurality of data acquisition sub-components that respectively perform data acquisition according to different data sources.

3. The data quality management method based on programmable components according to claim 1 is characterized in that: The data evaluation component includes a uniqueness evaluation subcomponent, an integrity evaluation subcomponent, an accuracy evaluation subcomponent, a consistency evaluation subcomponent, a relevance evaluation subcomponent, and a timeliness evaluation subcomponent.

4. The data quality management method based on programmable components according to claim 1 is characterized in that: The data modification component includes a cross-validation method data correction subcomponent for correcting erroneous data and missing data, and a similarity comparison method data removal subcomponent for removing redundant data.

5. The data quality management method based on programmable components according to claim 1 is characterized in that: The data analysis component includes a regression analysis subcomponent, a factor analysis subcomponent, a fishbone diagram analysis subcomponent, a Pareto analysis subcomponent, and a matrix data analysis subcomponent.

6. The data quality management method based on programmable components according to claim 1 is characterized in that: The data storage component includes a plurality of independent data storage sub-components.

Images