A power data aggregation processing method based on configurable operators

Abstract

The application relates to the technical field of electric power data aggregation processing, in particular to an electric power data aggregation processing method based on configurable operators, which comprises the following steps: firstly, acquiring and standardizing original data from multiple business systems; secondly, dragging, connecting and configuring preset filtering, grouping, aggregation and connection operators through a visual interface to form a graphical data processing flow; thirdly, parsing and generating the flow into an executable aggregation calculation model; fourthly, loading basic data and executing the model to obtain a structured result aggregated according to preset dimensions; and finally, outputting a business evaluation report based on the result and configured evaluation rules; the application encapsulates data processing logic into graphical configurable operators, so that business personnel can directly and flexibly combine and adjust the data aggregation flow without coding, effectively solving the problems of rigid flow and slow response caused by technical threshold, and improving the agility and ease of use of electric power data analysis.

Description

Technical Field

[0001] This invention relates to the field of power data aggregation and processing technology, and in particular to a power data aggregation and processing method based on configurable operators. Background Technology

[0002] In data analytics applications within the power industry, such as line loss calculation, load forecasting, and equipment condition assessment, it is typically necessary to integrate data from multiple independent systems, including electricity consumption data collection, production management, and marketing. These heterogeneous data sources mean that traditional data processing methods heavily rely on technical personnel writing dedicated ETL scripts or complex SQL code.

[0003] However, the construction and modification of data processing logic rely entirely on professional coding, which makes it impossible for analysts who are proficient in business rules to directly and quickly configure and adjust the data aggregation process.

[0004] When business needs change or new analytical dimensions need to be tried, developers must re-understand the requirements, modify and test the code. The whole process is slow to respond and has high communication costs, making data analysis and model iteration rigid and inflexible. Summary of the Invention

[0005] The purpose of this invention is to provide a power data aggregation processing method based on configurable operators, which solves the problem that analysts cannot directly and quickly define and adjust cross-system data aggregation logic due to technical barriers, resulting in rigid analysis processes and slow response.

[0006] To achieve the above objectives, the present invention provides a power data aggregation processing method based on configurable operators, comprising the following steps: The system acquires raw data from multiple heterogeneous power business systems and performs standardized transformation on the raw data according to a predefined power data standard model to generate a standardized basic data table. Based on the user's operation on the visual configuration interface, a graphical data processing flow consisting of multiple configurable sub-nodes connected by directed relationships is obtained, and the running parameters configured for each configurable sub-node are obtained. The graphical data processing flow and its running parameters are analyzed to generate a corresponding aggregation calculation model that can be executed by the computing engine; wherein, the aggregation calculation model defines the data processing logic sequence from the basic data table to the target aggregation result; Load the basic data table and execute the aggregation calculation model to obtain structured result data after aggregation calculation according to preset dimensions; Based on the structured result data and the evaluation rules configured for the evaluation operator nodes, the evaluation calculation is performed and the business evaluation results are output.

[0007] This process involves acquiring raw data from multiple heterogeneous power business systems and standardizing the raw data according to a predefined power data standard model to generate a standardized basic data table. Specifically, this includes: Obtain raw business data, including electricity consumption, load, and equipment status, from the electricity information collection system, production management system, and marketing system; The fields in the original business data are mapped to the corresponding fields in the power data standard model, and the data format is unified. The mapped data is cleaned to generate the standardized base data table.

[0008] The configurable operator nodes include at least four types: The filtering operator node has running parameters including logical conditional expressions for data filtering; The grouping operator node has one or more fields as the basis for grouping as its running parameters; Aggregate operator nodes, whose running parameters include at least one aggregate function type and its numerical field of action; The connection operator node has the following parameters: connection method and primary key and foreign key fields used to match data rows.

[0009] Specifically, the process of parsing the graphical data processing flow and its operating parameters to generate a corresponding aggregate computing model that can be executed by the computing engine includes: The graphical data processing flow is converted into a directed acyclic graph data structure, where nodes correspond to configured operator instances and edges correspond to the data flow between nodes; Traverse the directed acyclic graph and generate a series of task instructions that can be scheduled and executed by the computing engine, based on the operator type and running parameters corresponding to each node.

[0010] The step of loading the basic data table and executing the aggregation calculation model to obtain structured result data after aggregation calculation according to preset dimensions specifically includes: The distributed computing engine is invoked to distribute the task instructions in the aggregated computing model to multiple computing nodes; Each computing node executes its assigned tasks in parallel, performing filtering, grouping, aggregation, and join operations on the loaded basic data tables; The intermediate calculation results from each computing node are aggregated to generate the final structured result data.

[0011] The preset dimensions include at least one of the following: power supply area, line number, voltage level, or time period.

[0012] Specifically, the step of performing evaluation calculations and outputting business evaluation results based on the structured result data and the evaluation rules configured for the evaluation operator nodes includes: In the visual configuration interface, establish the data connection relationship between the output node of the aggregation calculation model and the evaluation operator node; Configure the calculation logic and threshold parameters of the business evaluation indicators for the evaluation operator node to form the evaluation rules; The structured result data is input into the evaluation operator node, triggering the evaluation rules to perform calculations, generating and outputting a business evaluation report containing quantitative results.

[0013] Specifically, after obtaining the graphical data processing flow based on user operations, and when executing the aggregation calculation model or the evaluation calculation, version management operations can be performed on the configuration information of the graphical data processing flow and the evaluation operator node. The version management operations include: saving configuration snapshots, recording version change information, supporting the query and retrospection of historical versions, and releasing specific versions as official running versions.

[0014] This invention presents a power data aggregation processing method based on configurable operators, transforming data processing logic from traditional coding implementation to graphical, parameterized configuration implementation. By encapsulating core data processing functions (such as filtering, grouping, aggregation, and joining) into standardized, independently configurable operators and providing an intuitive visual orchestration interface, business analysts can independently construct data aggregation processes that meet complex business needs through simple drag-and-drop, joining, and parameter settings, without requiring professional programming skills. When analysis requirements change, users only need to adjust the corresponding operators or parameters in the graphical interface, and the system can quickly generate a new executable model, greatly improving the agility and responsiveness of process refactoring. This method not only reduces reliance on professional technical personnel and shortens the data preparation cycle but also promotes the accumulation, reuse, and unified management of data processing logic through operator standardization. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0016] Figure 1 This is a flowchart of the power data aggregation processing method based on configurable operators of the present invention. Detailed Implementation

[0017] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0018] Please see Figure 1 ,in Figure 1 This is a flowchart of a power data aggregation processing method based on configurable operators.

[0019] This invention provides a power data aggregation processing method based on configurable operators, comprising the following steps: S1: Obtain raw data from multiple heterogeneous power business systems, and standardize the raw data according to a predefined power data standard model to generate a standardized basic data table.

[0020] Specifically, S11: Obtain raw business data including electricity consumption, load, and equipment status from the electricity information collection system, production management system, and marketing system.

[0021] In this embodiment, this step is executed through a pre-configured data adapter interface. For the electricity information collection system, production management system, and marketing system, dedicated adapters matching their data protocols and interface specifications are configured respectively. For example, for the electricity information collection system providing real-time data streams based on MQTT, a corresponding MQTT client adapter is configured for subscription and listening; for the production management system providing a Web Service query interface, a corresponding service call adapter is configured. These adapters automatically obtain raw business data streams or data snapshots containing key attributes such as electricity consumption, load, and equipment status from various business sources according to a predetermined scheduling plan (such as periodic polling) or in response to specific events.

[0022] S12: Map the fields in the original business data to the corresponding fields in the power data standard model, and unify the data format.

[0023] In this implementation, this step is executed based on a predefined power data standard model, which is a metadata framework containing core entities, attributes, data types, and business meanings in the power field. The acquired raw business data, with each field mapped to a standard field defined in the standard model according to its field name, data type, and contextual semantics, through predefined mapping rules or manual verification. For example, the "Total Positive Active Power" field from source A and the "Electricity Sales" field from source B are both mapped to the "Total Electricity Consumption" field in the standard model. Simultaneously, data format standardization is performed, including converting time information to a standard timestamp format, uniformly converting power values ​​to values ​​in kilowatts (kW), uniformly converting voltage values ​​to values ​​in kilovolts (kV), and standardizing the numerical precision.

[0024] S13: Clean the mapped data to generate the standardized basic data table.

[0025] In this implementation, this step is executed based on the cleaning rules defined by the configured power business knowledge base. The data, after field mapping and format unification, undergoes quality cleaning. This process includes: verifying whether measured values ​​are within a reasonable physical range based on equipment technical parameter files, and marking or correcting any identified out-of-limit data according to rules; filling missing points in key continuous sequence data (such as load curves) using interpolation algorithms based on nearby time data; identifying and eliminating completely duplicate data records based on business keys (such as "equipment ID-timestamp"); and uniformly marking missing values ​​in non-critical fields as null values. After the above transformation and cleaning process, a standardized basic data table with a unified structure and controllable quality is finally generated, serving as the sole reliable data source for subsequent graphical process configuration and aggregation calculations.

[0026] S2: Based on the user's operation on the visual configuration interface, obtain the graphical data processing flow consisting of multiple configurable sub-nodes connected by directed relationships, and obtain the running parameters configured for each configurable sub-node.

[0027] Specifically, the configurable operator nodes include at least the following four types: The filtering operator node has running parameters including logical conditional expressions for data filtering; The grouping operator node has one or more fields as the basis for grouping as its running parameters; Aggregate operator nodes, whose running parameters include at least one aggregate function type and its numerical field of action; The connection operator node has the following parameters: connection method and primary key and foreign key fields used to match data rows.

[0028] In this embodiment, the visual configuration interface includes at least an operator selection area, a process orchestration area, and a parameter configuration area. Users can instantiate and place desired operator nodes from the operator library provided in the operator selection area by dragging and dropping. By establishing directed connections between the input and output ports of different operator nodes, users can intuitively define the data flow path and processing order between operators, thereby assembling a complete graphical data processing flow. When a user selects any operator node in the process orchestration area, the parameter configuration area will dynamically load and display the parameter configuration form corresponding to that node type. Users can configure the node's operating parameters through this form: for filtered operator nodes, configurations such as "Voltage > 220 AND Time BETWEEN '2023-01-01' AND..." Logical conditional expressions such as "2023-01-31" are used; for grouping operator nodes, one or more fields (such as "Power Supply Station" or "Line ID") are selected from the data field list as grouping identifiers; for aggregation operator nodes, aggregation functions such as summation and averaging are selected for specified numerical fields (such as "Electricity Consumption"); for connection operator nodes, the two data sources to be connected, the connection type (such as inner join or left join), and the combination of primary key and foreign key fields used for row matching are specified. The system responds to all the above configuration operations by the user in real time and stores the resulting graphical data processing flow containing complete topology and parameter information.

[0029] S3: Analyze the graphical data processing flow and its running parameters to generate a corresponding aggregation calculation model that can be executed by the calculation engine; wherein, the aggregation calculation model defines the data processing logic sequence from the basic data table to the target aggregation result.

[0030] Specifically, S31: The graphical data processing flow is converted into a directed acyclic graph data structure, where nodes correspond to configured operator instances and edges correspond to the data flow between nodes; S32: Traverse the directed acyclic graph and generate a series of task instructions that can be scheduled and executed by the computing engine based on the operator type and running parameters corresponding to each node.

[0031] In this embodiment, this step is executed through a process compilation module, which is responsible for translating the user-configured business-oriented visual process on the front end into computing tasks that the back-end computing engine can directly execute.

[0032] Specifically, in step S31, the process compilation module reads and parses the configuration information of the graphical data processing flow. It converts the topology formed by the user through dragging and connecting lines in the flow into an internal directed acyclic graph (DAG) data structure. In this graph, each operator node with configured parameters (e.g., a "filter operator" with specific filtering conditions) is mapped to a vertex in the graph, while the directed connections between nodes representing the data flow are mapped to edges connecting these vertices. This conversion process strips away all visual attributes related to the user interface, retaining only pure computational logic and data dependencies, and ensuring that there are no cyclic dependencies in the graph, thus guaranteeing that the generated processing logic can theoretically be executed sequentially without falling into an infinite loop.

[0033] Next, in step S32, the process compilation module performs a topological sorting traversal of the generated directed acyclic graph. For each vertex (i.e., operator instance) in the graph, the module "compiles" it into one or more atomic task instructions recognizable by the target computing engine (e.g., Spark, Flink, or a high-performance SQL engine) based on its corresponding operator type (e.g., filtering, grouping, aggregation, and joining) and the specific runtime parameters configured by the user for that node. For example, a filtering operator node configured with the condition "voltage > 220" will have its runtime parameters and operator type compiled into a filtering instruction fragment similar to the WHERE clause in SQL; an aggregation operator node configured to group by "line ID" and sum "electricity consumption" will be compiled into an instruction fragment containing GROUP BY and SUM operations. The process compilation module sets a strict execution order for these instruction fragments based on the dependencies defined by the edges in the graph, and serializes and packages all fragments to ultimately generate a complete, self-contained aggregation computing model. This model encodes a complete computational blueprint that starts with a standardized base data table, goes through a series of user-defined transformation steps, and finally outputs the target aggregated data result.

[0034] S4: Load the basic data table and execute the aggregation calculation model to obtain structured result data after aggregation calculation according to preset dimensions.

[0035] Specifically, S41: Invoke the distributed computing engine to distribute the task instructions in the aggregated computing model to multiple computing nodes; S42: Each computing node executes its assigned tasks in parallel, performing filtering, grouping, aggregation, and join operations on the loaded basic data table; S43: Summarize the intermediate calculation results of each computing node to generate the final structured result data.

[0036] Furthermore, the preset dimensions include at least one of the following: power supply area, line number, voltage level, or time period.

[0037] In this embodiment, the execution of this step is completed collaboratively by a distributed computing engine (such as Apache Spark or Flink) and a task scheduler. Specifically, in step S41, the task scheduler receives a serialized aggregation computing model generated by the process compilation module. The scheduler first parses the model and breaks down the serialized task instructions into multiple subtasks that can be executed independently or in stages, based on data dependencies and a preset parallelism strategy. Subsequently, the scheduler communicates with the cluster manager of the distributed computing engine to distribute these subtasks to multiple available computing nodes in the cluster. Each computing node obtains the task instruction package it needs to execute and the related data sharding information.

[0038] In step S42, each computing node independently loads its assigned data shard from the standardized base data table. Each node executes the computational logic in parallel in its local memory or storage according to the specific content of its task instruction package. For example, one node might be responsible for filtering and preliminary grouping data within a certain time range, while another node might be responsible for performing join operations on another portion of the data. All nodes follow the logic defined in the aggregation computation model, performing operations such as filtering, grouping, aggregation, and joining, and generating their respective intermediate computation results. This process fully utilizes the cluster's parallel computing capabilities, significantly improving processing speed.

[0039] Finally, in step S43, after all parallel tasks have been completed, the driver program of the distributed computing engine or the designated aggregation node collects the intermediate computing results generated by each computing node through the network. Based on the final aggregation logic defined in the aggregation computing model (e.g., globally merging the results of distributed group aggregation), the engine performs further reduction, sorting, or merging operations on these intermediate results. This ultimately generates structured result data that conforms to preset dimensions (such as by region, route, or time period) and aggregation requirements. This result data will be persistently stored in a specified database or file system for subsequent analysis, visualization, or as input for step S5.

[0040] S5: Based on the structured result data and the evaluation rules configured for the evaluation operator node, perform evaluation calculations and output the business evaluation results.

[0041] In this embodiment, this step is executed by graphically integrating and executing the data aggregation process and business evaluation logic in the visual configuration interface, thereby forming an end-to-end configurable model from data processing to business insight.

[0042] Specifically, S51: In the visualization configuration interface, establish the data connection relationship between the output node of the aggregation calculation model and the evaluation operator node.

[0043] In this embodiment, the user connects the output port of the "structured result data" node (i.e., the final node of the aggregation calculation process) representing the final output of step S4 to the input port of a newly added evaluation operator node via a directed connection line in the process orchestration area of ​​the visual configuration interface. This operation establishes the data source for the evaluation calculation at the graphical level, meaning that the evaluation process will be based on the data generated by the aforementioned aggregation calculation and summarized by dimension.

[0044] S52: Configure the calculation logic and threshold parameters of the business evaluation indicators for the evaluation operator node to form the evaluation rules.

[0045] In this implementation, when a user selects an evaluation operator node in the canvas, a dedicated evaluation configuration form will appear in the parameter configuration area. The user defines specific evaluation rules in this form, which mainly includes two aspects: First, the calculation logic of business evaluation indicators. For example, the calculation formula for the line loss rate indicator can be defined as "(Power Supply - Power Sales) / Power Supply × 100%", where the user needs to configure and map the variables in the formula (such as 'Power Supply' and 'Power Sales') to specific fields in the upstream structured result data; or the average load rate indicator can be defined with the formula "(Average Load / Rated Capacity) × 100%". Second, threshold parameters and judgment conditions can be configured for each indicator. For example, a threshold of "5.0%" can be set for the line loss rate, and a rule can be set: "If the line loss rate > 5.0%, the evaluation conclusion is marked as 'abnormal'; otherwise, it is 'normal'." Multi-level thresholds and associated alarm actions can also be configured to form complex business rules.

[0046] S53: Input the structured result data into the evaluation operator node, trigger the evaluation rules to perform calculations, and generate and output a business evaluation report containing quantitative results.

[0047] In this implementation, when a user triggers the execution command for the entire integrated model, the execution engine first completes steps S1 to S4, generating structured result data. This data stream is then used as input to drive the execution of the evaluation operator node. This node loads the configured evaluation rules and applies calculation formulas line by line to each record in the input data (such as aggregated data for each power supply area or each line within a specified period) to obtain quantitative results for the indicators. Based on threshold conditions, it automatically generates status judgments or level assessments. Finally, all calculation results are organized, summarized, and formatted into a structured business evaluation report. This report includes data tables, a summary of key indicators, a list of anomalies, and visual charts. It can be directly displayed in the interface, supports interactive analysis, and can also be automatically exported as a standard document or pushed to other business systems. In this way, business personnel can independently complete the complete and dynamic assembly and execution of the business evaluation model from data aggregation rules through graphical configuration, lowering the technical threshold for data analysis.

[0048] Furthermore, after obtaining the graphical data processing flow based on user operations, and when executing the aggregation calculation model or the evaluation calculation, version management operations can be performed on the configuration information of the graphical data processing flow and the evaluation operator node. The version management operations include: saving configuration snapshots, recording version change information, supporting the query and retrospection of historical versions, and releasing specific versions as official running versions.

[0049] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A method for power data aggregation processing based on configurable operators, characterized in that, Includes the following steps: The system acquires raw data from multiple heterogeneous power business systems and performs standardized transformation on the raw data according to a predefined power data standard model to generate a standardized basic data table. Based on the user's operation on the visual configuration interface, a graphical data processing flow consisting of multiple configurable sub-nodes connected by directed relationships is obtained, and the running parameters configured for each configurable sub-node are obtained. The graphical data processing flow and its running parameters are analyzed to generate a corresponding aggregation calculation model that can be executed by the computing engine; wherein, the aggregation calculation model defines the data processing logic sequence from the basic data table to the target aggregation result; Load the basic data table and execute the aggregation calculation model to obtain structured result data after aggregation calculation according to preset dimensions; Based on the structured result data and the evaluation rules configured for the evaluation operator nodes, the evaluation calculation is performed and the business evaluation results are output.

2. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, Acquire raw data from multiple heterogeneous power business systems, and standardize the raw data according to a predefined power data standard model to generate a standardized basic data table, specifically including: Obtain raw business data, including electricity consumption, load, and equipment status, from the electricity information collection system, production management system, and marketing system; The fields in the original business data are mapped to the corresponding fields in the power data standard model, and the data format is unified. The mapped data is cleaned to generate the standardized base data table.

3. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, The configurable operator nodes include at least four types: The filtering operator node has running parameters including logical conditional expressions for data filtering; The grouping operator node has one or more fields as the basis for grouping as its running parameters; Aggregate operator nodes, whose running parameters include at least one aggregate function type and its numerical field of action; The connection operator node has the following parameters: connection method and primary key and foreign key fields used to match data rows.

4. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, The graphical data processing flow and its operating parameters are analyzed to generate a corresponding aggregate computing model that can be executed by the computing engine, specifically including: The graphical data processing flow is converted into a directed acyclic graph data structure, where nodes correspond to configured operator instances and edges correspond to the data flow between nodes; Traverse the directed acyclic graph and generate a series of task instructions that can be scheduled and executed by the computing engine, based on the operator type and running parameters corresponding to each node.

5. The power data aggregation processing method based on configurable operators as described in claim 4, characterized in that, The process of loading the basic data table and executing the aggregation calculation model to obtain structured result data after aggregation calculation according to preset dimensions specifically includes: The distributed computing engine is invoked to distribute the task instructions in the aggregated computing model to multiple computing nodes; Each computing node executes its assigned tasks in parallel, performing filtering, grouping, aggregation, and join operations on the loaded basic data tables; The intermediate calculation results from each computing node are aggregated to generate the final structured result data.

6. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, The preset dimensions include at least one of the following: power supply area, line number, voltage level, or time period.

7. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, The process of performing evaluation calculations and outputting business evaluation results based on the structured result data and the evaluation rules configured for the evaluation operator nodes specifically includes: In the visual configuration interface, establish the data connection relationship between the output node of the aggregation calculation model and the evaluation operator node; Configure the calculation logic and threshold parameters of the business evaluation indicators for the evaluation operator node to form the evaluation rules; The structured result data is input into the evaluation operator node, triggering the evaluation rules to perform calculations, generating and outputting a business evaluation report containing quantitative results.

8. The power data aggregation processing method based on configurable operators as described in claim 1, characterized in that, After obtaining the graphical data processing flow based on user operations, and when executing the aggregation calculation model or the evaluation calculation, version management operations can be performed on the configuration information of the graphical data processing flow and the evaluation operator node. The version management operations include: saving configuration snapshots, recording version change information, supporting the query and retrospection of historical versions, and releasing specific versions as official running versions.

Images