Electric power big data quality collaborative governance method

Abstract

The invention relates to an electric power big data quality collaborative governance method, which comprises the following steps of: 1, establishing a data quality check rule by configuring data source information and a built-in rule and running a script; step 2, based on the online data quality checking rule in the step 1, establishing an automatic scheduling task, and obtaining problem data; step 3, one-key order sending and problem data treatment are carried out according to the problem data obtained by the scheduling task in the step 2; and step 4, configuring a quality report according to the governance result in the step 3, regularly generating the data governance condition in the current period in a report form, and reversely promoting the data governance efficiency. According to the method, the quality condition of the existing data is monitored through the automatic scheduling task, and problem data is found in time for treatment.

Description

Technical Field

[0001] The present invention belongs to the technical field of power data quality verification, and relates to a method for collaborative governance of big data quality, and in particular to a method for collaborative governance of power big data quality. Background Art

[0002] With the deepening of digital transformation and the continuous advancement of the construction of new power systems, data has become increasingly prominent as a core strategic resource for enterprises. As a power grid company, power data is an important indicator of social and economic development and people's livelihood security. Promoting data sharing and integration and tapping the potential value of data have become important measures and effective ways to enhance the core competitiveness of enterprises and serve the high-quality development of society. Data quality is the basis for ensuring the effectiveness of data application. Data quality is an indicator that describes the value content of data. Just like the gold content of ore, the quality of data determines the value of data.

[0003] In recent years, based on the Tianjin Company's data middle platform, data quality verification, data quality assessment and other tasks have been effectively carried out. A series of means and methods have been initially formed to collect data quality rules, collect data quality issues, conduct data quality governance, and promote source data quality improvement based on the data middle platform. A semi-automated data quality verification script has been realized, and data quality verification reports are regularly released based on the data sharing innovation application platform to promote the company's overall data quality improvement work.

[0004] With the rapid development of smart grid construction, the types of power data are complex and diverse, and the data quality is uneven. The data integration and implementation process within the power industry encounters problems such as cumbersome data transmission routes, long transmission time, lack of effective monitoring and problem handling mechanisms during the transmission process, and data accuracy and timeliness that cannot meet the requirements of line loss calculation. First, the data quality verification work is only in the hands of data middle-end operation technicians. Although the verification script has achieved a semi-automatic mode, there is no effective platform for unified display and collection; second, the measurement data inspection needs to log in to a separate platform, and there is no unified tool to execute data Quality verification; third, the data quality rules are hidden in the background database, and there is no systematic and systematic rule catalog display capability; fourth, the data quality rules have not formed an effective data quality rule collection path, and data quality is only mined and defined from a technical perspective, and the business personnel are not closely linked, and there is no effective collection path for business perspective rules; fifth, the data quality collaborative governance workflow is still in offline mode, and online process means such as autonomous dispatching and leaving traces at processing nodes have not been formed; sixth, data quality verification has not formed a unified analysis module, and it is impossible to implement the idea of ​​"data managing data, using data to govern data" to govern data.

[0005] Therefore, the present invention proposes a method for collaborative management of power big data quality, which can solve the above problems.

[0006] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method for collaborative management of power big data quality. By automatically scheduling tasks to monitor the quality of existing data, problem data can be discovered and managed in a timely manner. Based on the regularly generated quality reports, the management of problem data and the reasons for failure to manage them in a timely manner can be mastered, thereby adjusting the data management method and improving data management efficiency.

[0008] The present invention solves the practical problem by adopting the following technical solutions:

[0009] A method for collaborative governance of power big data quality, comprising the following steps:

[0010] Step 1: Run the script to establish data quality verification rules by configuring data source information and built-in rules;

[0011] Step 2: Based on the data quality verification rules put online in step 1, establish an automatic scheduling task to obtain problem data;

[0012] Step 3: One-click dispatch and problem data management based on the problem data obtained from the scheduling task in step 2;

[0013] Step 4: Based on the governance results of step 3, configure quality reports and regularly generate data governance status for the current period in the form of reports to reversely promote data governance efficiency.

[0014] Moreover, the specific steps of step 1 include:

[0015] Step 1.1: Configure data source information and built-in rules to execute data quality verification rule scripts;

[0016] Step 1.2: Based on the data source information and built-in rules configured in step 1.1, configure the data quality verification rule script;

[0017] Step 1.3: Configure the result table based on the data quality check rule script in step 1.2 to complete the establishment of data quality check rules;

[0018] Moreover, the specific steps of step 2 include:

[0019] Step 2.1: Based on the data quality verification rules launched in step 1, configure the scheduling task;

[0020] Step 2.2: Based on the scheduling task configured in 2.1, start the scheduled scheduling task through JAVA multi-threading technology to obtain the problem data;

[0021] Moreover, the specific steps of step 3 include:

[0022] Step 3.1: Dispatch the task with one click based on the problem data obtained from the task dispatching in step 2;

[0023] Step 3.2: Perform data management based on the work order generated by the one-click dispatch in 3.1:

[0024] Moreover, the specific steps of step 4 include:

[0025] Step 4.1: Based on the governance results of step 3, establish data quality verification rules;

[0026] Step 4.2: Based on the data quality verification rules established in 4.1, configure the quality report template;

[0027] Step 4.3: Create a quality report based on the data quality verification rules established in 4.1 and the quality report template configured in 4.2;

[0028] Step 4.4: Subscribe and view the quality report established in step 4.3.

[0029] Advantages and beneficial effects of the present invention:

[0030] The present invention provides a method for collaborative governance of power big data quality to solve the key and difficult points of daily data quality management. First, it absorbs the normal work results of data quality verification in the data center, forms an online management method for the existing offline work mode, and fully solidifies the existing semi-automatic verification scripts in the platform. Technical personnel will fully carry out data quality verification work online, solidify work processes, verification scripts, task execution, etc., to facilitate existing complex work and avoid work interruptions caused by personnel changes; second, it supports measurement data verification capabilities, supports the collection and execution of measurement data rules, and unifies the company's data quality verification entrance; third, it forms a unified display of data quality rules, provides a display of all rules, and effectively sorts similar rules, rules for the same table, and rules for the same field, so that data management personnel and data application personnel can fully understand First, understand the company's data quality rules; fourth, provide a data quality rule collection module, business personnel can fill in data quality rules and provide data quality verification scripts through the page; fifth, build a data quality governance collaboration process. When data quality problems are found, the dispatch process can be called up with one click, connecting upstream and downstream processing personnel, quickly locating the cause of quality problems, and timely managing data quality, and providing a one-click re-inspection function; sixth, build a data quality operation analysis module, regularly monitor the execution of data quality verification scripts, analyze the application of data quality rules, and precipitate frequently called data quality rules, so as to provide basic data for the company to build authoritative data quality rules and form the idea of ​​"data management data, data governance data". BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a process flow chart of the present invention. DETAILED DESCRIPTION

[0032] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0033] A collaborative governance method for power big data quality, such as Figure 1 As shown, the following steps are included:

[0034] Step 1: Run the script to establish data quality verification rules by configuring data source information and built-in rules;

[0035] The specific steps of step 1 include:

[0036] Step 1.1: Configure data source information and built-in rules to execute data quality verification rule scripts;

[0037] Step 1.2: Based on the data source information and built-in rules configured in step 1.1, configure the data quality verification rule script;

[0038] Step 1.3: Configure the result table based on the data quality check rule script in step 1.2 to complete the establishment of data quality check rules;

[0039] In this embodiment, the step 1.1 is specifically as follows:

[0040] Configure the data source connection based on the database name, data source type, host IP, database name, port, user name, and password, and verify whether the configuration information is correct through JDBC technology;

[0041] In this embodiment, in step 1.1, the data source is MySQL, Oracle, pg, HBase, Hive, Python, Kafka, Hetu;

[0042] Configure built-in rules based on built-in rule name, check type (field check / condition check), database type, number of check fields, target value, and check logic content;

[0043] If you need to configure a script that queries the number of entries based on conditions, you must first configure the field check logic (count(*)) and condition check logic (${column}=?). The resulting built-in rule script framework is "select count(*)from${tables}where${column}=?", where multiple condition check logic and field check logic can be configured.

[0044] In this embodiment, the step 1.2 is specifically as follows:

[0045] Configure rule scripts based on script type (custom or built-in rules), built-in rules, data source, schema, verification table name, verification field, and verification script.

[0046] When Custom is selected for the script type, the rule script can be written manually.

[0047] When you select a built-in rule, you need to configure the script according to the script framework generated by the built-in rule. For example, when the built-in rule script framework is "select count(*)from${tables}where${column}=?", you need to select one or more check tables, check fields, and field target values. The tool will automatically generate a check script based on the configuration information.

[0048] For example, if you select t_auth_user as the check table, IS_USEABLE as the check field, and the target value is Y, the check script generated by the tool is "select count(*) from t_auth_user where IS_USEABLE='Y'".

[0049] In this embodiment, the step 1.3 is specifically as follows:

[0050] Based on the rule script configured in 1.2, the result data is obtained through JDBC technology to verify whether the rule script is configured correctly;

[0051] After the rules are launched, the script is parsed to dynamically create a result table for storing the returned data set, and personalized configuration of the result table data is provided to finally complete the establishment of data quality verification rules.

[0052] Step 2: Based on the data quality verification rules put online in step 1, establish an automatic scheduling task to obtain problem data;

[0053] The specific steps of step 2 include:

[0054] Step 2.1: Based on the data quality verification rules launched in step 1, configure the scheduling task;

[0055] Step 2.2: Based on the scheduling task configured in 2.1, start the scheduled scheduling task through JAVA multi-threading technology to obtain the problem data;

[0056] In this embodiment, the step 2.1 is specifically as follows:

[0057] Based on the data quality verification rules launched in step 1, establish an automatic scheduling task by configuring the execution cycle, execution node, first execution time, and whether to automatically dispatch orders;

[0058] In this embodiment, in step 2.1, the data quality verification rules are divided into unidirectional rules and bidirectional rules. The unidirectional rule has built-in checking logic in the verification script, and the data returned by the script are all problematic data; the bidirectional rule needs to compare the data in the two libraries, verify whether the data is consistent by checking the MD5 value or the number of entries, and return the verification result.

[0059] In this embodiment, the step 2.2 is specifically as follows:

[0060] Based on the scheduling task configured in 2.1, start the scheduled scheduling task through JAVA multi-threading technology, periodically execute the verification script, or choose to execute the scheduling task manually; during the execution of the scheduling task, the execution start and end time, execution duration, and execution status will be recorded, and the data source will be connected through JDBC technology to obtain the script return result set and save it to the rule result table.

[0061] For example: the data quality verification rules established in step 1 (each rule will have a number, assuming it is 202410220001, the fields after script parsing are cust_id, cust_name, cust_no), the result table is named T_RESULT_202410220001, the fields in the table are cust_id, cust_name, cust_no, is_blank (whether it is whitelisted, whitelisted data does not need to be managed), fk_task_result_method (task execution record mark), fk_method_script (script mark), after each task is executed, the original data returned by the script will be saved in the T_RESULT_202410220001 table.

[0062] Step 3: One-click dispatch and problem data management based on the problem data obtained from the scheduling task in step 2;

[0063] The specific steps of step 3 include:

[0064] Step 3.1: Dispatch the task with one click based on the problem data obtained from the task dispatching in step 2;

[0065] Step 3.2: Perform data management based on the work order generated by the one-click dispatch in 3.1:

[0066] In this embodiment, the step 3.1 is specifically as follows:

[0067] Based on the problem data obtained by scheduling tasks in step 2, dispatch work orders by configuring the type of issue, urgency, issuing user and other information;

[0068] If the dispatch type is an access problem or a model problem, the dispatch will be sent to the designated person;

[0069] If the distribution type is data owner system, the problematic data will be automatically distributed to the data owner based on the data identified by the data owner;

[0070] For example, the problem checked by the data quality verification rule is "the operating capacity of the customer's electricity consumption information is inconsistent with the total operating capacity of the equipment". In the problem result, there is a data with the primary key of "elec_cons_cust_id=122305110016285". The production owner of this data in the electricity user table identified by the data owner is Zhao Hui, so this problematic data will be automatically assigned to Zhao Hui.

[0071] In this embodiment, the step 3.2 is specifically as follows:

[0072] Data governance is performed based on the work order generated by 3.1 One-click order dispatch. When the dispatch type is an access problem or a model problem, the dispatching user checks whether the problem data has been completely managed after managing the problem data. If the management is completed, it can be submitted to the work order dispatcher to close the work order;

[0073] When the release type is data master, the scheduling task will automatically determine whether the historical work order has been managed the next time it is executed, and close the managed work order;

[0074] Each work order will record the dispatch time, issuing user, work order management status and other information to facilitate real-time tracking of data management progress.

[0075] Step 4: Based on the governance results of step 3, configure quality reports and regularly generate data governance status for the current period in the form of reports to reversely promote data governance efficiency.

[0076] The specific steps of step 4 include:

[0077] Step 4.1: Based on the governance results of step 3, establish data quality verification rules;

[0078] Step 4.2: Based on the data quality verification rules established in 4.1, configure the quality report template;

[0079] Step 4.3: Create a quality report based on the data quality verification rules established in 4.1 and the quality report template configured in 4.2;

[0080] Step 4.4: Subscribe and view the quality report established in step 4.3.

[0081] In this embodiment, the step 4.1 is specifically as follows:

[0082] Based on the governance results of step 3, establish data quality verification rules with the purpose of "quality report" and put the rules online to obtain data governance status regularly;

[0083] For example, to create a rule for completing data governance of issues this month, the corresponding rule script is "select count(*)as govern_count from t_work_order where month(o.govern_time)=month(current_date)and year(o.govern_time)=year(current_date)".

[0084] In this embodiment, the step 4.2 is specifically as follows:

[0085] Based on the data quality verification rules established in 4.1, configure the quality report template. The template needs to configure the report title and support segment configuration. Each paragraph is displayed in the form of a sub-template. Text, charts, and key data can be configured in the paragraph. Dynamic replacement is supported;

[0086] For example, the data quality check rule created in 4.1 (each rule will have a number, assuming it is 202410120001) is to obtain the number of tables checked this month. The corresponding return value field is CHECK_TABLE_COUNT. If you need to obtain this number in the paragraph, you can change the text to "${202410120001_CHECK_TABLE_COUNT} tables checked this month". When generating the report, ${202410120001_CHECK_TABLE_COUNT} will be automatically replaced with the result data of the data quality check rule execution.

[0087] In this embodiment, the step 4.3 is specifically as follows:

[0088] Based on the data quality verification rules established in 4.1 and the quality report template configured in 4.2, you can configure the cycle for generating quality reports. The scheduled task will automatically generate quality reports regularly according to the configured cycle.

[0089] In this embodiment, the step 4.4 is specifically as follows:

[0090] Based on the quality report established in 4.3, the company's business personnel can subscribe to and export the generated quality report. If the quality report is subscribed, the generated quality report will be pushed regularly for business personnel to view and export;

[0091] Through the content of the quality report, the company's business personnel can clearly identify the work orders, data governance personnel, scenarios, etc. involved in the ungoverned data, and promptly adjust the data quality rules in step 1 and promote the governance status of step 3 according to the content of the quality report, thereby reversely promoting data governance efficiency.

[0092] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for collaborative management of power big data quality, characterized by: The steps include: Step 1: Run the script to establish data quality verification rules by configuring data source information and built-in rules; Step 2: Based on the data quality verification rules put online in step 1, establish an automatic scheduling task to obtain problem data; Step 3: One-click dispatch and problem data management based on the problem data obtained from the scheduling task in step 2; Step 4: Based on the governance results of step 3, configure quality reports and regularly generate data governance status for the current period in the form of reports to reversely promote data governance efficiency.

2. According to claim 1, a method for collaborative management of power big data quality is characterized by: The specific steps of step 1 include: Step 1.1: Configure data source information and built-in rules to execute data quality verification rule scripts; Step 1.2: Based on the data source information and built-in rules configured in step 1.1, configure the data quality verification rule script; Step 1.3: Configure the result table based on the data quality check rule script in step 1.2 to complete the establishment of data quality check rules.

3. The method for collaborative management of power big data quality according to claim 1 is characterized by: The specific steps of step 2 include: Step 2.1: Based on the data quality verification rules launched in step 1, configure the scheduling task; Step 2.2: Based on the scheduling task configured in 2.1, start the scheduled scheduling task through JAVA multi-threading technology to obtain the problem data.

4. The method for collaborative management of power big data quality according to claim 1 is characterized by: The specific steps of step 3 include: Step 3.1: Dispatch the task with one click based on the problem data obtained from the task dispatching step 2; Step 3.2: Perform data management based on the work orders generated by the one-click dispatch in 3.

1.

5. The method for collaborative management of power big data quality according to claim 1 is characterized by: The specific steps of step 4 include: Step 4.1: Based on the governance results of step 3, establish data quality verification rules; Step 4.2: Based on the data quality verification rules established in 4.1, configure the quality report template; Step 4.3: Create a quality report based on the data quality verification rules established in 4.1 and the quality report template configured in 4.2; Step 4.4: Subscribe and view the quality report established in step 4.3.

Images