Method for implementing power scheduling mass data parallel computing and storage medium

Abstract

This invention discloses a method and storage medium for parallel computing of massive data in power dispatching. The method performs lexical analysis on computing requests, analyzes different phrases and interdependencies in the computing requests to obtain a lexical analysis table, combines the analyzed lexical analysis table into a multi-branch tree to form a syntax analysis tree, then transforms the syntax analysis tree into a sequential dependency graph of object operations to form an object computation graph, and finally uses a vectorized computing engine to transform the object computation graph into a series of vector primitive operations to form a vector computation graph, and executes these vector primitive operations on a multi-core processor, thereby improving the parallel efficiency of computing massive, heterogeneous data in power dispatching.

Description

Technical Field

[0001] This invention relates to a method and storage medium for parallel computing of massive data in power dispatching, and more particularly to a data computing method and storage medium that converts massive data computing requests into a series of vector primitive operations in multiple steps and supports execution on multi-core processors. Background Technology

[0002] Electricity is a vital material foundation for modern society's production, economic development, and daily life. With advancements in science and technology, human demand for electricity grows annually, and the informatization of China's power industry has also made significant progress. The rapid construction and application of renewable energy generation capacity, such as wind and solar power, which exhibit strong random fluctuations, has led to a surge in power dispatch data resources. This data is characterized by its large volume, complex structure, and dispersed storage. To meet the real-time requirements of renewable energy output forecasting, day-ahead user forecasting, and load flow analysis, there is an urgent need for high-performance processing and computing technologies for massive real-time data to achieve high-level integration and efficient processing of data information at different levels, granularities, and time periods, both internal and external.

[0003] To address the existing problems, it is necessary to conduct research on real-time and near-real-time data characteristics and processing, taking into account the characteristics of power dispatching operations and data features, in order to achieve rapid querying and calculation of massive amounts of terminal meter data under high concurrency, and improve the efficient processing of different types of real-time data related to power dispatching. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and storage medium for parallel computing of massive data in power dispatching. This method performs lexical analysis on computing requests, analyzes different phrases and interdependencies of the computing requests to obtain a lexical analysis table, combines the analyzed lexical analysis table into a multi-branch tree to form a syntax analysis tree, then transforms the syntax analysis tree into a sequential dependency graph of object operations to form an object computation graph, and finally transforms the object computation graph into a series of vector primitive operations through a vectorized computing engine to form a vector computation graph, and allows these vector primitive operations to be executed on a multi-core processor, thereby improving the parallel efficiency of data computing.

[0005] This invention for scheduling massive data computation includes the following steps:

[0006] Step 101, Receive request:

[0007] Obtain the request to be queried and calculated, as input, including:

[0008] The request records three types of information: the object to be calculated, the calculation conditions, and the return result. The calculation object is the object ID, the calculation conditions include object associations and association conditions, and the return result includes object attributes and calculation requirements.

[0009] Step 102, Lexical Analysis Table:

[0010] Lexical analysis is performed on the input request to obtain fields, objects, conditions, calculations, and results, including:

[0011] Lexical analysis is performed on the input requests, and the calculation requests are broken down and refined into five types: object, field, condition, calculation, and result. The breakdown of each type is explained below:

[0012]

[0013] Step 103, Parse Tree:

[0014] Based on the required output, a parse tree is constructed from bottom to top to obtain important steps and information of the computation process, including:

[0015] Based on an Abstract Syntax Tree (AST) decoder, a bottom-up inverted tree is recursively constructed, with the root at the top and the leaves at the bottom. The linear order of the syntactic structure tree is as follows: First, a group of words forming a part of a sentence at a branch node of the tree is called a syntactic structural component, and the bottom of the tree contains individual words. Second, the syntactic component structure tree has a hierarchical structure, and the groups of words that make up the syntactic structural components are represented by the level they appear at in the tree. Third, syntactic structural components have syntactic categories. A component includes all smaller components at the lower levels in the structure tree. The branches of the tree are called nodes, the syntactic categories associated with the nodes are called labels, and individual words are called leaves.

[0016] The types of parsing tree nodes are described in the table below:

[0017]

[0018] Step 104, Object Computation Graph:

[0019] Based on the lexical analysis table, the parsing tree is transformed into a coarse-grained object computation graph, including:

[0020] First, record the object's data table address and the object's attributes. Following the syntax tree structure, and using a top-down parsing approach, convert the parse tree into a directed acyclic object computation graph. The construction steps are as follows:

[0021] 1) Construct query nodes and compute nodes separately to form an object set;

[0022] 2) Sub-query nodes or fields are constructed sequentially under the query node to form an object filter set; sub-compute nodes or join nodes are constructed sequentially under the compute node to form a join condition set.

[0023] 3) Sub-connection nodes or evaluation nodes are constructed sequentially under the connection node to form the evaluation set;

[0024] 4) Following the connection order of the parsing tree, complete the connection of each set and determine the order of their sequence to form a directed acyclic object computation graph.

[0025] Step 105, Vector computation graph:

[0026] Transforming an object computation graph into a vector computation graph includes:

[0027] The transformation from an object computation graph to a vector computation graph does not involve actual data computation. Each object computation node corresponds to one or more vector nodes, and each vector node consists of one or more vector primitive operations. Each vector primitive operation has a corresponding backend service function interface, which executes the operation by calling the corresponding function. Since the data objects processed by vector primitive operations are vectors, except for the read operation which has only one output vector and no input vector, each vector operation has at least one input vector and one output vector. Through a certain organization, vector primitive operations must possess relatively complex functions to complete computational tasks such as filtering, joining, and aggregating complex data.

[0028] The operation types of vector primitives are described in the table below:

[0029]

[0030] Step 106, Calculation results:

[0031] Perform vector operations and finally output the calculation results, including:

[0032] The above steps do not involve actual data manipulation or computation. Actual execution and computation only occur after the data is converted into vector operations and submitted to the backend system. During vector execution, vectors with a logical relationship and order must be executed sequentially, while vectors without such relationship or order can be executed in parallel. Vector execution is encapsulated into parallelizable instructions by the backend system, enabling parallel computation on multi-core CPUs.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the steps of the method of the present invention.

[0034] Beneficial effects of the present invention

[0035] This invention accelerates the performance of massive data processing by transforming massive data computation requests into a series of vector primitive operations in multiple steps and enabling these vector primitive operations to be computed in parallel on multi-core CPUs. In particular, it can significantly improve the processing efficiency of scheduling multi-type and multi-scale massive data for complex calculations of power dispatching data. Attached Figure Description

[0036] Figure 1 This is the overall flowchart of the present invention.

[0037] Figure 2 This is a lexical analysis diagram of the present invention.

[0038] Figure 3 This is a schematic diagram of the parsing tree of the present invention.

[0039] Figure 4 This is a computational diagram of the object of the present invention.

[0040] Figure 5 This is a vector calculation diagram for the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0042] Example 1

[0043] The specific calculation steps of this invention are as follows:

[0044] Step 101: Receive Request

[0045] Retrieve the object to be calculated and the requirements from the client request. The request contains three categories: object, condition, and return result. An example request is shown below:

[0046]

[0047] Step 102: Lexical Analysis Table

[0048] Lexical analysis is performed on the input request, and the calculation request is broken down and refined according to five types: object, field, condition, calculation, and result. The results of the breakdown and refinement are as follows: Figure 2 As shown, the object dictionary table is queried based on the object's ID to obtain the fields of each object, and the calculation conditions in the returned results are obtained based on lexical analysis.

[0049] Step 103: Parse Tree

[0050] Based on the Abstract Syntax Tree (AST) decoder, a bottom-up inverted parse tree is recursively constructed, with the root at the top and the leaves at the bottom. Based on the required output [a2, b2, c2, average, sum, max], the parse tree is constructed from bottom to top to obtain the important steps and information of the computation process. The constructed parse tree is as follows: Figure 3As shown, the parsing tree includes the Qry query node and the Cal compute node on the left. The Qry node can be directly refined to the three leaf nodes of a2, b2, and c3. The Cal node includes two Con join nodes. The Con nodes are further refined to the three evaluation nodes of Ave, Sum, and Max. Finally, the evaluation nodes are joined with the leaf nodes of the output fields.

[0051] Step 104: Object Computation Graph

[0052] First, record the data table addresses of objects X, Y, and Z, as well as the attributes of the three objects, according to... Figure 3 The parsing tree structure employs a top-down parsing approach, transforming the parsing tree into a directed acyclic object computation graph. Figure 4 The specific construction steps are explained in detail below:

[0053] 1) Construct query nodes and compute nodes respectively to form an object set [X′,Y′,Z′];

[0054] 2) Sub-query nodes or fields are constructed sequentially under the query node to form object filter set 1 [(x.a1=2),(y.b3=5),(y.b2=z.c1)] and object filter set 2 [(x.a1=y.b2)]. Sub-compute nodes or join nodes are constructed sequentially under the compute node to form join condition set [(X′join Y′),(X′join Y′join Z′)];

[0055] 3) Construct child join nodes or evaluation nodes under each join node in sequence to form the evaluation set [AVE′, SUM′, MAX′];

[0056] 4) Following the connection order of the parse tree, complete the connection of each set and determine its order, forming a directed acyclic object computation graph, such as... Figure 4 As shown.

[0057] Step 105: Vector computation graph

[0058] according to Figure 4 Object computation graph, construction Figure 5 The vector computation graph is constructed by first building three vector nodes (labeled as 1), and then constructing two filtering operations as conditional mask MK operations. Next, two vector nodes (labeled as 2) are constructed, followed by two connection vector nodes (labeled as 3 and 4 respectively). The required vector primitive operation type is connection JN operation. After vector node 4, a vector grouping GP operation is constructed to assist in the calculation of the evaluation node. After vector node 5, a three-vector aggregation AP operation is constructed, followed by a vector merging MG operation to merge the three vectors (labeled as 6). Finally, vector node 8 is constructed, corresponding to the vector sorting SR operation.

[0059] Step 106: Calculation Results

[0060] The above steps do not involve actual data manipulation or computation. Actual execution and computation only occur after the data is converted into vector operations and submitted to the backend system for execution. Vector operations without sequential relationships or order can be executed in parallel; for example, conditional mask (MK) operations and aggregation (AP) operations can form parallel instructions and can be computed in parallel on multi-core CPUs.

[0061] The evaluation and testing results of this embodiment are as follows:

[0062] Through the above steps, the client's computation request is transformed into a series of vector primitive operations in multiple steps. Among them, vector primitive operations such as MK operation and AP operation can be computed in parallel on multi-core CPUs. This can significantly improve performance and shorten computation time for processing and computing massive amounts of power dispatch data.

Claims

1. A method for parallel computing of massive data in power dispatching, characterized in that, This method improves the parallel efficiency of data computation by performing lexical analysis on computation requests, analyzing different phrases and dependencies within the requests to obtain a lexical analysis table. The analyzed lexical analysis tables are then combined into a multi-way tree to form a syntax analysis tree. This syntax analysis tree is further transformed into a sequence dependency graph of object operations, forming an object computation graph. Finally, a vectorized computation engine converts the object computation graph into a series of vector primitive operations, forming a vector computation graph. These vector primitive operations are then executed on a multi-core processor. Specifically, the method includes the following steps: Step 101, Receive Request: Obtain the request to be queried and calculated as input; Step 102, Lexical Analysis Table: Perform lexical analysis on the input request to obtain fields, objects, conditions, calculations, and results; Step 103, Parsing Tree: Based on the required output, construct the parsing tree from bottom to top to obtain important steps and information in the computation process; including: recursively constructing a bottom-up inverted tree based on the Abstract Syntax Tree (AST) decoder; the linear order of its grammar tree structure is as follows: First, at the branch nodes of the tree, there is a group of words that form a part of a sentence, called a grammatical structural component, and at the bottom of the tree are individual words; Second, the grammatical component tree structure has a hierarchical structure, and the phrases that make up the grammatical components are represented by the levels that appear in the tree. Third, grammatical structural components have syntactic categories. A component includes all smaller components below it in the tree structure hierarchy. The branches of the tree are called nodes, the syntactic categories associated with the nodes are called tags, and individual words are called leaves. Step 104, Object Computation Graph: Based on the lexical analysis table, the parsing tree is transformed into a coarse-grained object computation graph; this includes: first, recording the data table address of the object and the object's attributes; then, following the parsing tree structure, using a top-down parsing approach, converting the parsing tree into a directed acyclic object computation graph. The construction steps are as follows: 1) Construct query nodes and compute nodes separately to form an object set; 2) Sub-query nodes or fields are constructed sequentially under the query node to form an object filter set; sub-compute nodes or join nodes are constructed sequentially under the compute node to form a join condition set. 3) Sub-connection nodes or evaluation nodes are sequentially constructed under the connection node to form the evaluation set; 4) Following the connection order of the parse tree, complete the connection of each set and determine its order, forming a directed acyclic object computation graph. Step 105, Vector Computation Graph: Transforming the object computation graph into a vector computation graph; including: the conversion from object computation graph to vector computation graph does not involve actual data computation; each object computation node corresponds to one or more vector nodes; each vector node has one or more vector primitive operations; each vector primitive operation has a corresponding backend service function interface; the backend executes the operation by calling the corresponding function interface; the data objects processed by vector primitive operations are vectors, so except for the data read operation which has no input vector but only one output vector, each vector operation has at least one input vector and one output vector; the vector primitive operations must have relatively complex functions to complete the filtering, joining, and aggregation calculations of complex data through a certain organization; Step 106, Calculation result: Perform vector operations and finally output the calculation result.

2. The method for parallel computing of massive data in power dispatching as described in claim 1, characterized in that, Step 101 includes: The request records three types of information: the object to be calculated, the calculation conditions, and the return result. The calculation object is the object ID, the calculation conditions include object associations and association conditions, and the return result includes object attributes and calculation requirements.

3. The method for parallel computing of massive power dispatch data as described in claim 1, characterized in that, Step 102 includes: Lexical analysis is performed on the input requests, and the calculation requests are broken down and refined according to five types: object, field, condition, calculation, and result.

4. The method for parallel computing of massive power dispatch data as described in claim 1, characterized in that: Steps 101-106 do not involve actual data manipulation or calculation. Actual execution and calculation only occur after the data is converted into vector operations and submitted to the backend, where the backend execution system performs the vector operations.

5. The method for parallel computing of massive power dispatch data as described in claim 4, characterized in that: When executing vectors, vectors that are related and ordered need to be executed sequentially; vectors that are not related and ordered are executed in parallel; the execution of vectors is encapsulated into instructions that can be executed in parallel by the background system and computed in parallel on a multi-core CPU.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program can be executed by a processor to implement the steps of the method for parallel computation of massive data in power dispatching as described in any one of claims 1-5.

Images