Data quality closed-loop management method based on PDCA theory

Abstract

The invention discloses a data quality closed-loop management method based on a PDCA theory, and the method comprises the steps: building a data quality rule library, carrying out the key sorting of quality rules according to data application and data sharing scenes, and formulating a data quality stage and a long-term target. A data quality problem management module is established, timely monitoring, online feedback follow-up and review of quality problems are achieved, timely rectification and closing of the data quality problems are guaranteed, and a data quality management tool or platform is established; through the method, comprehensive monitoring of various data of an enterprise can be realized, early warning can be provided for data application and data sharing problems in time, reason analysis is facilitated, the enterprise data quality management level is improved, and a good data environment is provided for the enterprise to realize digital intelligence; according to the method, comprehensive management of the data quality of the source sticking layer in the data lake and the data quality of the data center is achieved, hour-level and day-level problems can be issued and follow up according to the importance degree of data problems, and the timeliness of data quality problem processing is greatly guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality management, and in particular to a data quality closed-loop management method based on PDCA theory. Background Art

[0002] At present, the data quality management of most power generation companies only stays at the institutional norms level or has simple data quality monitoring, which mainly realizes the quality management of enterprise management data; as a power generation company, there are many production units and sites, and the quality problems of production data are difficult to follow up and improve in time. A simple production problem may take a long time, and the communication information only stays offline. The causes of the problem cannot be summarized to form a knowledge base, which makes it difficult to provide effective support for the overall improvement of production data management quality. Problems reported in various links of data reporting, data application or data consumption are often handled offline. When problems are encountered and then handled, there is no systematic quality control mechanism. The handling of quality problems is relatively delayed and the processing process is long. Therefore, it is necessary to design a data quality closed-loop management method based on PDCA theory to solve the above problems. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a data quality closed-loop management method based on the PDCA theory, so as to solve the problems in the prior art that quality problems of production data are difficult to effectively follow up and improve, the timeliness of quality problem improvements is extremely poor, and the causes of the problems cannot form a knowledge base; it has the characteristics of being able to transform passive discovery into active monitoring, while making data quality processing online, and timely understanding the processing progress of data problems.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: The data quality closed-loop management method based on the PDCA theory includes: Establish a data quality rule base, prioritize the sorting out of quality rules based on data application and data sharing scenarios, and formulate data quality phase and long-term goals; Establish a data quality problem management module to achieve timely monitoring of quality problems, online feedback follow-up and review, and ensure timely rectification and closure of data quality problems; Build data quality management tools or platforms.

[0005] Preferably, the data quality rule base includes a source layer rule base and a data center rule base.

[0006] Preferably, the source layer rule base covers the accessed management data and production data to verify the integrity, consistency, accuracy and timeliness of the data. The importance of the rules is medium or low, and the rectification period is in days. Generally, rectification is completed within 2-3 days. The source layer rule base includes the power generation or installed capacity of each station is not 0 or empty, and the active power of the station has no change for a long time / 24 hours.

[0007] Preferably, the data center rule base covers data operation monitoring and business indicator data to achieve validity and accuracy verification of operations and data. The rule importance is high, and the rectification period is in hours. Generally, rectification is completed within 4-8 hours.

[0008] Furthermore, the quality rules of the source layer are collected through data asset inventory and configured according to the concerns of data production users; the quality rules of the data warehouse are mainly configured by summarizing the execution alarm information of all development jobs and according to the indicators or data attributes that data consumption users are concerned about.

[0009] Preferably, establishing a data quality problem management module includes data monitoring and verification, and the data includes source layer data and data center data: The source layer data is divided into management data and production data, of which management data is collected and entered into the data lake. After entering the data lake, the table fields of management data are checked daily according to the configured quality rules, and problems that do not comply with the rules are recorded in the system dimension. Production data is checked daily according to the configured quality rules for measurement point data, and problem records are formed in the site dimension.

[0010] Data operation monitoring conducts alarm monitoring on the operation of scheduling operations on an hourly basis according to the configured quality rules. The operation of scheduling operations includes the collection, processing and services of the data center. Business indicator data conducts abnormal monitoring on the key indicator data of the core data application of power production on an hourly basis. When the indicator data does not meet the established threshold or rules, an alarm is issued.

[0011] Preferably, data monitoring and verification include standardization, completeness, accuracy, consistency, uniqueness and timeliness testing; management data is verified for the integrity, consistency and timeliness of key attributes of the business system; production data is verified for the integrity and timeliness of high-frequency time series data such as power generation, power, and water conditions.

[0012] Preferably, according to the data quality closed-loop management method based on PDCA theory of claim 5, it is characterized in that the problem records are divided into high-level quality problems, medium-level quality problems and low-level quality problems, and the data application is divided into attention levels according to user roles and user scopes during the data application construction period: Quality issues with a high level of concern are those arising from data applications that target company-level leaders or have a user base that is the entire company. Quality issues with a medium level of concern are quality issues arising from data applications targeting department leaders or with a user scope at the department level; Quality issues with a low level of concern are quality issues arising from data applications that are directed at business managers or implementation personnel and have a user scope of group level.

[0013] Furthermore, quality rules are divided into high, medium and low levels according to the degree of impact of data problems. If it directly affects the homepage of important applications, it is classified as high level; if it affects the secondary / lower pages of data applications, it is classified as medium level; if it basically has no direct impact on important data applications but affects other non-emergency businesses, it is classified as low level.

[0014] Preferably, for high-level quality problems, the system will immediately push prompts and issue a list of problem data, and the source end will contact and feedback immediately; For medium-level quality issues, if the abnormal problem data persists for 24 hours, a prompt will be pushed and a list of problem data will be issued. The source end will feedback the processing results within 48 hours. In the case of low-level quality issues, if the abnormal problem data persists for 48 hours, a prompt will be pushed and a list of problem data will be issued, and the source end will feedback the processing results within 48 hours.

[0015] Preferably, the establishment of a data quality problem management module includes problem feedback and review. After the problem is issued, the data quality management program triggers the rule re-run according to the feedback plan rectification time to realize data review. If the rectification is not made on time, notification reminders will continue to be sent, and the problem status will be changed to postponed and not rectified. The quality report will be notified later; specifically, it includes: Data quality problem management: including scalable configuration of problem detail fields, batch delivery of quality problems, automatic tracking and review of quality problems, problem cause summary, and manual closing or restarting of quality problems; When quality issues are issued in batches, production data is issued according to the site dimension, and management data is issued according to the system dimension; Data quality problem feedback: including quality problem details viewing and quality problem feedback; quality problem feedback includes the cause of the problem and the planned rectification time; Data quality problem feedback review: Review the cause of the problem and the time for rectification reported by the source. If the review is passed, proceed to the next step of review.

[0016] Preferably, a data quality management tool or platform is built; the data quality management tool or platform implements online management including configuration of data quality rules, data inspection, issue issuance and rectification and review.

[0017] The beneficial effects of the data quality closed-loop management method based on the PDCA theory provided by the present invention are as follows: 1. The present invention is implemented through the data quality management method and tool, which can realize comprehensive monitoring of various types of enterprise data, provide early warning for data application and data sharing problems in a timely manner, facilitate cause analysis, improve the enterprise data quality management level, and provide a good data environment for enterprises to realize digitalization; 2. The present invention realizes the comprehensive management of data quality at the source layer and data center in the data lake. According to the importance of data problems, it can realize the issuance and follow-up of problems at the hourly and daily levels, which greatly ensures the timeliness of data quality problem handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a schematic diagram of the process framework according to the present invention. DETAILED DESCRIPTION

[0019] Embodiment 1: like Figure 1 As shown in the figure, the closed-loop data quality management method based on the PDCA theory includes: Establish a data quality rule base, prioritize the sorting out of quality rules based on data application and data sharing scenarios, and formulate data quality phase and long-term goals; Establish a data quality problem management module to achieve timely monitoring of quality problems, online feedback follow-up and review, and ensure timely rectification and closure of data quality problems; Build data quality management tools or platforms.

[0020] Preferably, the data quality rule base includes a source layer rule base and a data center rule base.

[0021] Preferably, the source layer rule base covers the accessed management data and production data to verify the integrity, consistency, accuracy and timeliness of the data. The importance of the rules is medium or low, and the rectification period is in days. Generally, rectification is completed within 2-3 days. The source layer rule base includes the power generation or installed capacity of each station is not 0 or empty, and the active power of the station has no change for a long time / 24 hours.

[0022] Preferably, the data center rule base covers data operation monitoring and business indicator data to achieve validity and accuracy verification of operations and data. The rule importance is high, and the rectification period is in hours. Generally, rectification is completed within 4-8 hours.

[0023] Furthermore, the quality rules of the source layer are collected through data asset inventory and configured according to the concerns of data production users; the quality rules of the data warehouse are mainly configured by summarizing the execution alarm information of all development jobs and according to the indicators or data attributes that data consumption users are concerned about.

[0024] Preferably, establishing a data quality problem management module includes data monitoring and verification, and the data includes source layer data and data center data: The source layer data is divided into management data and production data, of which management data is collected and entered into the data lake. After entering the data lake, the table fields of management data are checked daily according to the configured quality rules, and problems that do not comply with the rules are recorded in the system dimension. Production data is checked daily according to the configured quality rules for measurement point data, and problem records are formed in the site dimension.

[0025] Data operation monitoring conducts alarm monitoring on the operation of scheduling operations on an hourly basis according to the configured quality rules. The operation of scheduling operations includes the collection, processing and services of the data center. Business indicator data conducts abnormal monitoring on the key indicator data of the core data application of power production on an hourly basis. When the indicator data does not meet the established threshold or rules, an alarm is issued.

[0026] Preferably, data monitoring and verification include standardization, completeness, accuracy, consistency, uniqueness and timeliness testing; management data is verified for the integrity, consistency and timeliness of key attributes of the business system; production data is verified for the integrity and timeliness of high-frequency time series data such as power generation, power, and water conditions.

[0027] Preferably, according to the data quality closed-loop management method based on PDCA theory of claim 5, it is characterized in that the problem records are divided into high-level quality problems, medium-level quality problems and low-level quality problems, and the data application is divided into attention levels according to user roles and user scopes during the data application construction period: Quality issues with a high level of concern are those arising from data applications that target company-level leaders or have a user base that is the entire company. Quality issues with a medium level of concern are quality issues arising from data applications targeting department leaders or with a user scope at the department level; Quality issues with a low level of concern are quality issues arising from data applications that are directed at business managers or implementation personnel and have a user scope of group level.

[0028] Furthermore, quality rules are divided into high, medium and low levels according to the degree of impact of data problems. If it directly affects the homepage of important applications, it is classified as high level; if it affects the secondary / lower pages of data applications, it is classified as medium level; if it basically has no direct impact on important data applications but affects other non-emergency businesses, it is classified as low level.

[0029] Preferably, for high-level quality problems, the system will immediately push prompts and issue a list of problem data, and the source end will contact and feedback immediately; For medium-level quality issues, if the abnormal problem data persists for 24 hours, a prompt will be pushed and a list of problem data will be issued. The source end will feedback the processing results within 48 hours. In the case of low-level quality issues, if the abnormal problem data persists for 48 hours, a prompt will be pushed and a list of problem data will be issued, and the source end will feedback the processing results within 48 hours.

[0030] Preferably, the establishment of a data quality problem management module includes problem feedback and review. After the problem is issued, the data quality management program triggers the rule re-run according to the feedback plan rectification time to realize data review. If the rectification is not made on time, notification reminders will continue to be sent, and the problem status will be changed to postponed and not rectified. The quality report will be notified later; specifically, it includes: Data quality problem management: including scalable configuration of problem detail fields, batch delivery of quality problems, automatic tracking and review of quality problems, problem cause summary, and manual closing or restarting of quality problems; When quality issues are issued in batches, production data is issued according to the site dimension, and management data is issued according to the system dimension; Data quality problem feedback: including quality problem details viewing and quality problem feedback; quality problem feedback includes the cause of the problem and the planned rectification time; Data quality problem feedback review: Review the cause of the problem and the time for rectification reported by the source. If the review is passed, proceed to the next step of review.

[0031] Preferably, a data quality management tool or platform is built; the data quality management tool or platform implements online management including configuration of data quality rules, data inspection, issue issuance and rectification and review.

[0032] Embodiment 2: Preferably, data monitoring and verification include standardization, completeness, accuracy, consistency, uniqueness and timeliness, i.e., the "six data quality properties" detection. Management data is verified for the integrity, consistency and timeliness of key attributes of the business system; production data is verified for the integrity and timeliness of high-frequency time series data such as power generation, power, and water conditions. The "six data quality properties" detection method is as follows: Standardization: The verification data must comply with the relevant data standards, data models, business rules, etc. of power production, such as whether the data items are stored in non-standard formats, whether the data types comply with the agreed value list, data accuracy, etc.; Integrity: Check whether the required data items are empty, for example, the power generation table of a station or unit is empty or has no data at certain time points.

[0033] Accuracy: Check whether the data item exceeds the normal value range, for example, the active power value of a unit is negative and far less than 0.

[0034] Consistency: Verify whether an indicator is consistent in multiple locations. For example, the comprehensive portal and the group's big data platform all have the power generation data of the Three Gorges Power Plant. It is necessary to verify whether the power generation data between multiple platforms is consistent.

[0035] Uniqueness: Check whether the code, name or value of the business object is consistent. For example, whether the organization is unique, whether there are multiple power generation data at the same site at the same time, etc.

[0036] Timeliness: Verify whether the expired data in business management has been changed in time. For example, an employee has retired, but the change has not been made in time in the unified certification; or a station has been sold, but the daily power generation is still being counted.

[0037] Embodiment three: According to the six characteristics of data quality and based on business management priorities, relevant data quality rules are identified and formulated for management data and production data respectively. Some examples of rules are shown in Table 1 below: Serial number Rule Description Rule Category type Level of concern 1 The daily power generation of the station cannot be 0, null, or no data Completeness Management high 2 The Internet power cannot be 0, null, or no data Completeness Management Low 3 The daily power generation of the station should not be less than 0 accuracy Management high 4 The Internet power should not be less than 0 accuracy Management Low 5 The daily power generation of the station should not exceed the power generation of full load (rated capacity) operation (installed capacity*24*1.2) accuracy Management high 6 The installed capacity of the station that has reported the power generation cannot be 0 or null Completeness Management middle 7 Daily average irradiance and daily average wind speed cannot be empty Completeness Management middle 8 The daily average irradiance and daily average wind speed cannot be negative and do not exceed the historical maximum value*5--Find the basis / historical data to set the threshold accuracy Management middle 9 The station's [daily power generation] should be the same as the cumulative value of [unit power generation] consistency Management Low 10 The data of key measuring points (water level, flow, power, wind speed, irradiance) have been interrupted and not updated for a long time (currently set at 24 hours). Completeness Centralized control high 11 The average active power value of the station is less than 0, and the total power value is greater than the installed capacity*1.2, for more than 4 hours accuracy Centralized control high 12 The active power of the station remains unchanged for a long time (24 hours) accuracy Centralized control middle 13 The station resources (wind speed, irradiance) remain unchanged for a long time (24 hours) accuracy Centralized control middle 14 Inflow / outflow flow, upstream / downstream water level remain unchanged for a long time (24 hours) accuracy Centralized control high 15 The proportion of units in alarm operation at new energy stations exceeds 30% accuracy Centralized control middle 16 The proportion of fault-related shutdown units at new energy stations exceeds 30% accuracy Centralized control middle 17 The proportion of units with communication interruptions at new energy stations exceeds 30% Completeness Centralized control middle Table 1: Data quality rules table.

[0038] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. The closed-loop data quality management method based on PDCA theory is characterized by: include: Establish a data quality rule base, prioritize the sorting out of quality rules based on data application and data sharing scenarios, and formulate data quality phase and long-term goals; Establish a data quality problem management module to achieve timely monitoring of quality problems, online feedback follow-up and review, and ensure timely rectification and closure of data quality problems; Build data quality management tools or platforms.

2. The data quality closed-loop management method based on PDCA theory according to claim 1 is characterized in that: The data quality rule base includes the source layer rule base and the data center rule base.

3. The data quality closed-loop management method based on PDCA theory according to claim 2 is characterized in that: The source layer rule library covers the accessed management data and production data to verify the integrity, consistency, accuracy and timeliness of the data. The importance of the rules is medium or low, and the rectification period is in days.

4. The data quality closed-loop management method based on PDCA theory according to claim 2 is characterized in that: The data center rule base covers data operation monitoring and business indicator data, realizing the validity and accuracy verification of operations and data.

5. The data quality closed-loop management method based on PDCA theory according to claim 1 is characterized in that: Establishing a data quality problem management module includes data monitoring and verification. The data includes source layer data and data center data: The data at the source layer is divided into management data and production data. Management data is collected and entered into the data lake. After entering the data lake, the table fields of management data are checked daily according to the configuration quality rules, and problems that do not meet the rules are recorded according to the system dimension. Production data verifies the measurement point data according to the configuration quality rules every day, and forms problem records according to the station dimension. Data operation monitoring conducts alarm monitoring on the operation of scheduling operations on an hourly basis according to the configured quality rules. The operation of scheduling operations includes the collection, processing and services of the data center. Business indicator data conducts abnormal monitoring on the key indicator data of the core data application of power production on an hourly basis. When the indicator data does not meet the established threshold or rules, an alarm is issued.

6. The data quality closed-loop management method based on PDCA theory according to claim 5 is characterized in that: Data monitoring and verification include testing of standardization, completeness, accuracy, consistency, uniqueness and timeliness. Management data is verified for the integrity, consistency and timeliness of key attributes of business systems. Production data is verified for the integrity and timeliness of high-frequency time series data such as power generation, power and water conditions.

7. The data quality closed-loop management method based on PDCA theory according to claim 5 is characterized in that: Problem records are divided into high-level quality issues, medium-level quality issues, and low-level quality issues. The attention levels are divided according to the user roles and user scopes of data applications during the data application construction period: Quality issues with a high level of concern are those arising from data applications that target company-level leaders or have a user base that is the entire company. Quality issues with a medium level of concern are quality issues arising from data applications targeting department leaders or with a user scope at the department level; Quality issues with a low level of concern are quality issues arising from data applications that are directed at business managers or implementation personnel and have a user scope of group level.

8. The data quality closed-loop management method based on PDCA theory according to claim 7 is characterized by: For high-level quality issues, the system will immediately push notifications and issue a list of problem data, and the source end will provide immediate feedback. For medium-level quality issues, if the abnormal problem data persists for 24 hours, a prompt will be pushed and a list of problem data will be issued. The source end will feedback the processing results within 48 hours. In the case of low-level quality issues, if the abnormal problem data persists for 48 hours, a prompt will be pushed and a list of problem data will be issued, and the source end will feedback the processing results within 48 hours.

9. The data quality closed-loop management method based on PDCA theory according to claim 1 is characterized by: Establish a data quality problem management module including problem feedback and review. After the problem is issued, the data quality management program triggers the rules to re-run according to the feedback plan rectification time to realize data review. If the rectification is not made on time, notification reminders will continue to be sent, and the problem status will be changed to postponed and not rectified. The quality report will be notified later. Specifically include: Data quality problem management: including scalable configuration of problem detail fields, batch delivery of quality problems, automatic tracking and review of quality problems, problem cause summary, and manual closing or restarting of quality problems; Data quality problem feedback: including quality problem details viewing and quality problem feedback; Data quality problem feedback review: Review the cause of the problem and the time for rectification reported by the source. If the review is passed, proceed to the next step of review.

10. The data quality closed-loop management method based on PDCA theory according to claim 1 is characterized by: Build data quality management tools or platforms; data quality management tools or platforms implement online management including configuration of data quality rules, data inspection, issue issuance, and rectification and review.

Images