Parallel task execution-based power marketing and sales power data quality inspection method, system and device, and storage medium

Abstract

The invention relates to the technical field of big data, in particular to a parallel task execution-based power marketing and selling power data quality inspection method, system and device and a storage medium. The data quality inspection method comprises the following steps: loading an inspection rule, a data source path and a parameter threshold; loading historical and current electricity sales data; data exception check, data repetition check, data missing check and data consistency check are carried out in parallel according to set check rules; and generating a detailed checking report. According to the method, a multi-task parallel processing technology is adopted, complex checking tasks are decomposed according to different rule categories, and then all kinds of checking tasks are executed at the same time through a parallel framework. In this way, the execution time of the checking task is remarkably shortened. Therefore, the large-scale data can be checked and fed back in a short time, so that the overall efficiency of data checking is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, system, device and storage medium for checking the quality of power marketing and sales electricity data based on parallel task execution. Background Art

[0002] With the digital transformation of the power industry, electricity sales data has become an important asset for power companies. These data cover multiple dimensions such as time, unit, industry classification, electricity consumption category, growth rate, etc., and are an important basis for power companies to make operational decisions, market forecasts, and service optimization. However, as the scale of data continues to expand, data quality issues have gradually emerged, including data missing, anomalies, and duplications. These problems not only increase operating costs, but may also lead to incorrect market judgments and decisions, further affecting customer experience and the market competitiveness of enterprises. Traditional data quality inspection methods mostly rely on static rule engines, which usually process inspection tasks serially and cannot efficiently cope with complex multi-source data environments.

[0003] Existing implementation solutions often focus on a single task and rely on set rules for data verification. For example, the rule engine can identify simple missing values ​​or format errors, but it lacks flexibility and adaptability when faced with complex data relationships in multiple dimensions. In addition, these solutions usually lack efficient parallel processing capabilities, resulting in a long response time for data checks, affecting the timeliness of decision-making.

[0004] In the field of power marketing analysis, existing technologies mainly rely on static rule engines for data quality checks. This approach has several obvious disadvantages:

[0005] (i) Inefficiency: Since inspection tasks are usually processed serially, the response time is long when faced with large amounts of data, and data quality issues cannot be discovered and resolved in a timely manner. As a result, decision makers may face delays in making important business decisions.

[0006] (II) Insufficient flexibility: Existing solutions lack the flexibility to adapt to complex data relationships. Power marketing data involves multiple dimensions (such as region, industry, and time), and a single task processing method is difficult to fully address multiple quality issues such as missing data, anomalies, and duplications. Summary of the invention

[0007] In view of the complexity and data quality issues of multi-source data in power marketing and sales analysis, this paper proposes an efficient data quality inspection and management technology solution. The details are as follows:

[0008] In a first aspect, a method for checking the quality of power sales data based on parallel task execution includes the following steps:

[0009] S100: Loading inspection rules, data source paths and parameter thresholds;

[0010] S200: Loading historical and current electricity sales data;

[0011] S300: Performing data anomaly check, data duplication check, data missing check and data consistency check in parallel according to the check rules set in S100;

[0012] S400: Generate a detailed inspection report.

[0013] Preferably, the data anomaly check includes the following process:

[0014] S301: for each industry or sector involved in a row, determine whether the data of power sales and cumulative power sales in the same period have a zero value; if so, record the corresponding zero-value industry or sector in S400, and then go to S302; otherwise, go directly to S302;

[0015] S302: for each industry or sector involved in a row, determine whether there are negative values ​​in the data of the electricity sales this month, the electricity sales in the same period, the cumulative electricity sales this year, and the cumulative electricity sales in the same period; if so, record the corresponding negative industry or sector in S400, and then go to S303; otherwise, go directly to S303;

[0016] S303: for each industry or sector involved in a row, determine whether the year-on-year growth rate and the cumulative year-on-year growth rate fluctuate within the set threshold range; if it exceeds the threshold range, record the corresponding industry or sector with excessive fluctuation in S400, and then go to S304; otherwise, directly go to S304;

[0017] S304: For each industry or sector involved in a row, determine whether its value is abnormally higher or lower than a set threshold range; if it exceeds the threshold range, record the industry or sector whose corresponding data is too large or too small in S400, and then end; otherwise, end directly.

[0018] Preferably, the data duplication check includes the following process:

[0019] Prevent duplicate data entry by comparing management units, industry names, and data dates.

[0020] Preferably, the data missing check includes the following process:

[0021] S321: for each industry or industry involved in a row, determine whether the basic data fields such as the management unit and industry name are missing; if missing, record the corresponding industry or industry with missing data fields in S400, and then go to S322; otherwise, directly go to S322;

[0022] S322: for each industry or sector involved in a row, determine whether the fields of power sales in the same period and cumulative power sales in the same period are missing; if they are missing, record the corresponding industry or sector with missing data fields in S400, and then go to S323; otherwise, go directly to S323;

[0023] S323: Determine whether the electricity sales data of all electricity sales industries under the same management unit are complete; if there are any missing or incomplete data, record the corresponding incomplete management unit in S400 and then end; otherwise, end directly.

[0024] Preferably, the data consistency check is based on a logical relationship to ensure that the electricity sales data under different categories are consistent with each other.

[0025] In a second aspect, a power marketing and sales electricity quantity data quality inspection system based on parallel task execution includes:

[0026] Configuration management module, used for unified management and initialization of inspection rules, data source paths, and parameter thresholds;

[0027] Data management module: used to load historical data and current data to be inspected from multi-source data platforms, store inspection results persistently, and ensure the traceability of the inspection process;

[0028] The task management module is used to divide the inspection task into several subtasks, track their status, and automatically handle abnormal situations during task execution;

[0029] The task scheduling module is used to allocate computing resources, coordinate the execution order and dependencies of tasks, and ensure efficient parallel processing of tasks on multiple nodes.

[0030] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for checking the quality of power marketing and sales electricity data based on parallel task execution is implemented.

[0031] In a fourth aspect, a non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for checking the quality of power marketing and sales electricity data based on parallel task execution.

[0032] The present invention uses multi-task parallel processing technology to decompose complex inspection tasks according to different rule categories, and then executes various inspection tasks simultaneously through a parallel framework. This method significantly shortens the execution time of inspection tasks. By disassembling inspection tasks and processing them in parallel, the time consumption of the entire inspection process is significantly reduced, so that large-scale data can be inspected and fed back in a shorter time, thereby effectively improving the overall efficiency of data inspection.

[0033] The present invention dynamically loads and initializes configuration parameters, including inspection rules, data source paths, threshold parameters, etc., before the inspection task is executed, and encodes the inspection rules into executable code. In this way, the inspection rules can be flexibly expanded and modified without large-scale adjustments or downtime maintenance of the system.

[0034] The present invention adopts an efficient parallel processing framework, which not only enables various inspection tasks to be executed simultaneously, but also optimizes the execution order of tasks through reasonable resource scheduling to avoid resource waste. This architecture ensures that the system can maintain high responsiveness and stability when processing large data sets.

[0035] Therefore, the present invention has the following significant advantages:

[0036] Improved efficiency: Through multi-task parallel processing, the time consumption of data inspection is significantly reduced, so that data quality issues can be identified and handled more quickly.

[0037] Accuracy and consistency: Dynamically load inspection rules to ensure accurate rule execution during the inspection process and ensure the reliability of inspection results.

[0038] High scalability: The system has good scalability and can flexibly adapt to data inspection needs of different scales according to the amount and complexity of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1-Figure 3 The electricity sales data of the present invention comes from 133 electricity sales industries in different power grids and provinces, including: electricity sales this month, electricity sales in the same period, year-on-year growth rate this month, cumulative electricity sales this year, cumulative electricity sales in the same period, and cumulative year-on-year growth rate attributes. Due to the large number of table rows, Table 1, Table 2, and Table 3 are connected in sequence.

[0040] Figure 4 A schematic flow chart of a method for checking the quality of power marketing and sales electricity volume data based on parallel task execution in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The electricity sales data involves multiple dimensions, including region, industry and time, which are intertwined to form complex relationships and dependencies. For example, the electricity sales data of the "primary industry", "secondary industry" and "tertiary industry", in addition to the year-on-year growth rate and cumulative year-on-year growth rate this month, other data, such as electricity sales this month, electricity sales in the same period, cumulative electricity sales this year, and cumulative electricity sales in the same period, must sum to the corresponding data item of "total electricity consumption in the whole society". This relationship reflects the mutual influence between industries and the importance of data consistency.

[0042] A method for checking the quality of power sales data based on parallel task execution includes the following steps:

[0043] S100: Loading inspection rules, data source paths and parameter thresholds;

[0044] S200: Loading historical and current electricity sales data;

[0045] S300: Perform data anomaly check, data duplication check, data missing check and data consistency check in parallel according to the check rules set in S100; specifically including:

[0046] S301: for each industry or sector involved in a row, determine whether the data of power sales and cumulative power sales in the same period have zero values; if so, record the corresponding zero-value industry or sector in S400, and then go to S302; otherwise, go directly to S302 to avoid erroneous data values ​​and ensure the rationality of growth rate calculation;

[0047] S302: for each industry or sector involved in a row, determine whether there are negative values ​​in the data of the electricity sales this month, the electricity sales in the same period, the cumulative electricity sales this year, and the cumulative electricity sales in the same period; if so, record the corresponding negative industry or sector in S400, and then go to S303; otherwise, go directly to S303; negative data may indicate that the data may be entered incorrectly or calculated abnormally;

[0048] S303: for each industry or sector involved in a row, determine whether the year-on-year growth rate and the cumulative year-on-year growth rate of this month fluctuate within a reasonable range within the set threshold range; if it exceeds the threshold range, record the corresponding industry or sector with excessive fluctuations in S400, and then go to S304; otherwise, go directly to S304; under normal circumstances, the growth rate will not have abnormal values ​​that are too high or too low, such as a growth rate of more than ±1000%, and an abnormal growth rate may indicate that the data is wrong;

[0049] The calculation formula for this month's year-on-year growth rate is:

[0050]

[0051] The cumulative electricity sales this year is obtained by adding the cumulative electricity sales of the previous month to the electricity sales of this month;

[0052] S304 Data extreme value check: for each industry or industry involved in a row, determine whether its value is abnormally higher or lower than a set threshold range; if it exceeds the threshold range, record the industry or industry whose corresponding data is too large or too small in S400, and then end; otherwise, end directly; the data table name data exceeding the threshold range may be entered incorrectly or calculated abnormally;

[0053] The data duplication check includes the following process: preventing duplicate data input by comparing the management unit, industry name, and data date; ensuring the uniqueness of each record;

[0054] The data missing check includes the following process:

[0055] S321: for each industry or industry involved in a row, determine whether the basic data fields such as the management unit and industry name are missing; if missing, record the corresponding industry or industry with missing data fields in S400, and then go to S322; otherwise, directly go to S322;

[0056] S322: for each industry or sector involved in a row, determine whether the fields of power sales in the same period and cumulative power sales in the same period are missing; if they are missing, record the corresponding industry or sector with missing data fields in S400, and then go to S323; otherwise, go directly to S323; it cannot be empty, and the missing of key data fields will directly affect the growth rate and trend analysis;

[0057] S323: Determine whether the power sales data of all power sales industries under the same management unit are complete; if there are any missing or incomplete data, record the corresponding incomplete management unit in S400 and then end; otherwise, end directly; missing data of any power sales industry will lead to data calculation errors for the entire industry;

[0058] The data consistency check is based on logical relationships to ensure that the electricity sales data under different categories are consistent with each other; for example, the total electricity sales of the whole society should be equal to the sum of the electricity sales of the primary industry, the secondary industry and the tertiary industry (except for the two columns of year-on-year growth rate and cumulative year-on-year growth rate);

[0059] S400: Generate a detailed inspection report.

[0060] The data quality inspection method mainly consists of several key components that work together to achieve efficient data processing:

[0061] Configuration management component: The configuration management module is used to uniformly manage and initialize the inspection rules, data source paths, parameter thresholds, etc. This component ensures the dynamic loading and updating of the inspection rule code, avoiding the inefficient operation of repeated coding each time the algorithm is executed. Through this mechanism, the rule set can be flexibly expanded to meet the data inspection needs in different scenarios.

[0062] Data management component: The data management component is responsible for loading historical data and current data to be inspected from multi-source data platforms, such as databases or file systems, to ensure data integrity and consistency. This component also supports persistent storage of inspection results to ensure traceability of the inspection process. This component can efficiently handle large-scale data loading and provide reliable data support.

[0063] Task management component: The task management component in the present invention automatically divides the inspection task into several subtasks and tracks their status. Through this mechanism, the inspection tasks can be executed in parallel as needed to ensure the maximum utilization of resources. The task management component is also responsible for real-time monitoring of the task execution status, automatically handling abnormal situations in task execution, and enhancing the stability of the algorithm.

[0064] Task scheduling component: The core innovation of the inspection algorithm is to use a parallel computing framework to schedule the execution of multiple inspection tasks. The task scheduling module is responsible for the reasonable allocation of computing resources, and through the load balancing optimization algorithm, coordinates the execution order and dependencies of tasks to ensure efficient parallel processing of tasks on multiple nodes. This component greatly improves the processing efficiency of the data inspection process.

Claims

1. A method for checking the quality of power sales data based on parallel task execution, characterized in that: The process includes the following: S100: Loading inspection rules, data source paths and parameter thresholds; S200: Loading historical and current electricity sales data; S300: Performing data anomaly check, data duplication check, data missing check and data consistency check in parallel according to the check rules set in S100; S400: Generate a detailed inspection report.

2. According to the method for checking the quality of power sales data based on parallel task execution of power marketing according to claim 1, it is characterized in that: The data anomaly check includes the following process: S301: for each industry or sector involved in a row, determine whether the data of power sales and cumulative power sales in the same period have a zero value; if so, record the corresponding zero-value industry or sector in S400, and then go to S302; otherwise, go directly to S302; S302: for each industry or sector involved in a row, determine whether there are negative values ​​in the data of the electricity sales this month, the electricity sales in the same period, the cumulative electricity sales this year, and the cumulative electricity sales in the same period; if so, record the corresponding negative industry or sector in S400, and then go to S303; otherwise, go directly to S303; S303: for each industry or sector involved in a row, determine whether the year-on-year growth rate and the cumulative year-on-year growth rate of this month fluctuate within a set threshold range; If it exceeds the threshold range, the corresponding industry or sector with excessive fluctuation is recorded in S400, and then the process goes to S304; otherwise, the process goes directly to S304; S304: For each industry or sector involved in a row, determine whether its value is abnormally higher or lower than a set threshold range; if it exceeds the threshold range, record the industry or sector whose corresponding data is too large or too small in S400, and then end; otherwise, end directly.

3. According to the method for checking the quality of power sales data based on parallel task execution of power marketing according to claim 1, it is characterized in that: The data duplication check prevents duplicate data entry by comparing management units, industry names, and data dates.

4. According to claim 1, a method for checking the quality of power sales data based on parallel task execution, characterized in that: The data missing check includes the following process: S321: for each industry or industry involved in a row, determine whether the basic data fields such as the management unit and industry name are missing; if missing, record the corresponding industry or industry with missing data fields in S400, and then go to S322; otherwise, directly go to S322; S322: for each industry or sector involved in a row, determine whether the fields of power sales in the same period and cumulative power sales in the same period are missing; if missing, record the corresponding industry or sector with missing data fields in S400, and then go to S323; Otherwise, go directly to S323; S323: Determine whether the electricity sales data of all electricity sales industries under the same management unit are complete; if there are any missing or incomplete data, record the corresponding incomplete management unit in S400 and then end; otherwise, end directly.

5. According to claim 1, a method for checking the quality of power sales data based on parallel task execution, characterized in that: The data consistency check is based on logical relationships to ensure that the electricity sales data under different categories are consistent with each other.

6. A power marketing and sales electricity data quality inspection system based on parallel task execution, characterized in that: include: Configuration management module, used for unified management and initialization of inspection rules, data source paths, and parameter thresholds; Data management module; Used to load historical data and current data to be inspected from multi-source data platforms, store inspection results persistently, and ensure the traceability of the inspection process; The task management module is used to divide the inspection task into several subtasks, track their status, and automatically handle abnormal situations during task execution; The task scheduling module is used to allocate computing resources, coordinate the execution order and dependencies of tasks, and ensure efficient parallel processing of tasks on multiple nodes.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for checking the quality of power marketing and sales electricity data based on parallel task execution as described in any one of claims 1 to 5 when executing the computer program.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for checking the quality of power marketing and sales electricity quantity data based on parallel task execution according to any one of claims 1 to 5 is implemented.

Images