A multi-dimensional data dynamic aggregation method for power project management and control

Abstract

The present application relates to the technical field of data dynamic aggregation, in particular to a multi-dimensional data dynamic aggregation method for power project management and control, comprising the following steps: extracting the material-to-goods quantity of a substation, the number of construction team members and voltage grade grouping identifier to form project request items, and establishing an abstract syntax feature sequence through lexical and syntactic analysis; traversing and sorting the data type nodes and project code nodes in the abstract syntax feature sequence, constructing a hierarchical relationship key-value pair, and generating a power project query syntax tree. The present application enables the numerical summary process to perform dynamic upgrade or direct summary according to the storage bit width risk, taking into account the aggregation accuracy and memory occupation control; the multi-dimensional aggregated numerical structured is read-only data snapshot, the dynamic reading pointer is merged and encapsulated as an index structure and assembled to the management and control memory pool, so that the project management and control side can directly locate the multi-dimensional aggregation result in the form of a cursor, improving the query response efficiency.

Description

Technical Field

[0001] This invention relates to the field of dynamic data aggregation technology, and in particular to a method for dynamic aggregation of multidimensional data for power project management. Background Technology

[0002] Dynamic data aggregation technology is designed for multi-source, heterogeneous, distributed, and real-time changing data objects. Through data extraction, field identification, grouping and mapping, local summarization, cross-node merging, and result encapsulation, it dynamically aggregates and calculates data content that is stored in a dispersed manner or generated continuously according to a unified business dimension.

[0003] Existing dynamic data aggregation technologies typically focus on field identification, grouping mapping, partial summarization, cross-node merging, and result encapsulation for multi-source, heterogeneous, distributed, and real-time changing data objects. In practice, they often use a unified business dimension as the aggregation entry point, lacking clear expression of the hierarchical relationship between field content and business codes. This can easily cause requests to remain at the field matching level, failing to reflect the execution dependencies between data type nodes, project code nodes, and group identifiers. When power projects involve multiple dimensions such as material arrival quantity, construction team personnel, and voltage level, if the aggregation entry point is only based on field names or fixed grouping dimensions, problems may arise such as unclear project code attribution under the same voltage level, duplicate data transmission across cities, and unstable group boundaries when merging partial summary results again. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multi-dimensional dynamic data aggregation method for power project management.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for dynamic aggregation of multi-dimensional data for power project management, comprising the following steps: The project request items are composed of the substation material arrival quantity, the number of construction team members entering the site, and the voltage level grouping identifier. An abstract syntactic feature sequence is established through lexical and syntactic parsing. The data type nodes and project code nodes in the abstract syntactic feature sequence are traversed and sorted to construct hierarchical relationship key-value pairs and generate a power project query syntax tree. Combining the multidimensional aggregation operators and each identifier in the power project query syntax tree, multidimensional cost estimates are calculated to establish push-down aggregation scheduling tasks; the multidimensional aggregation operators are decomposed and allocated to the city-level distribution node execution task queue to generate a city-level node aggregation execution scheme. The city-level node aggregation execution scheme is assigned to an independent data source node. The underlying stored data columns are read to perform local aggregation and generate a local summary value table. The voltage level grouping values ​​are extracted and mapped to the consistent hash routing ring addressing slot node through hash encoding. The data is then distributed through the data shuffling channel to generate shuffling distribution data blocks. The collected shuffled distribution data blocks are loaded into a memory buffer and a memory pipelined aggregation stream is established through multi-way merge sorting. Before performing the overlay aggregation operation, the estimated sum is checked according to the anti-overflow threshold condition. If the check fails, the data type is dynamically upgraded before aggregation. If the check passes, the aggregation is performed directly to generate multidimensional aggregated values. The multidimensional aggregated values ​​are structured into read-only data snapshots and dynamic read pointers are extracted, encapsulated into an index structure, and assembled into the management memory pool to generate a multidimensional project management cursor.

[0006] Preferably, the step of obtaining the abstract grammatical feature sequence is as follows: Extract the quantity of materials delivered to the substation, the number of personnel entering the construction team, and the voltage level grouping identifier. Read the field name, field value, field source location, and project code attribution mark corresponding to each item. Concatenate them according to the arrangement rule of field name first, field value in the middle, and project code attribution mark last. Remove empty fields, duplicate fields, and fields without project code attribution mark to obtain the project request item. The project request item is decomposed character by character according to the field name boundary, field value boundary, and project code attribution mark boundary. The start position, end position, and field attribution category of each character segment are recorded. The substation material arrival quantity, construction team personnel, and voltage level grouping identifier are marked as data retrieval identifiers. The start and end positions of the character segments are organized into character boundary information. The data retrieval identifiers and character boundary information are arranged in order according to the field attribution category to establish an abstract grammatical feature sequence.

[0007] Preferably, the steps for obtaining the power project query syntax tree are as follows: Based on the abstract grammatical feature sequence, retrieve the data type nodes corresponding to the substation material arrival quantity, the data type nodes corresponding to the number of construction team members entering the site, and the data type nodes corresponding to the voltage level group identifier. Continue to retrieve the project code nodes corresponding to each data type node. Perform traversal sorting according to the hierarchical order of root node, data type node, and project code node. Record the parent pointer, child pointer, and field mapping relationship between adjacent nodes. Extract the parent pointer, child pointer, and field mapping relationship as node association edge attributes. Construct the node association edge attributes into hierarchical relationship key-value pairs according to the hierarchical number and field belonging category to generate a power project query syntax tree.

[0008] Preferably, the step of obtaining the push-down aggregation scheduling task is as follows: Extract the multidimensional aggregation operator, substation material arrival quantity, construction team personnel, and voltage level grouping identifier from the power project query syntax tree. Read the parent node number, child node number, and hierarchy depth marker of the node where the multidimensional aggregation operator is located. Count the number of different grouping values ​​corresponding to the voltage level grouping identifier under the project coding node. Compare the hierarchy depth marker and the number of different grouping values ​​item by item. Calculate the multidimensional cost prediction value of the data source node hierarchy according to the cost accumulation order of the data source node hierarchy. Establish a push-down aggregation scheduling task.

[0009] Preferably, the steps for obtaining the city-level node aggregation execution plan are as follows: Read the multidimensional cost estimation value, data source node hierarchical relationship and cost accumulation order in the push-down aggregation scheduling task, determine the push-down priority of the multidimensional aggregation operator according to the multidimensional cost estimation value in ascending order, decompose the multidimensional aggregation operator into sub-tasks of substation material arrival quantity aggregation, construction team entry personnel aggregation, and voltage level group identification grouping, and allocate them to the corresponding city distribution nodes according to the data source node hierarchical relationship, and obtain the city distribution node execution task queue; Based on the task queue of the city-level distribution nodes, read the sub-tasks for aggregating substation material arrivals, construction team personnel, and voltage level grouping identifiers corresponding to each city-level distribution node. Verify the node execution order, field value range, and project code attribution flag for each sub-task. Write the decomposed multi-dimensional aggregation operator, substation material arrivals, construction team personnel, and voltage level grouping identifier into the task slots of the corresponding city-level distribution nodes to form a fixed mapping relationship between node task slots and field attribution categories, and generate a city-level node aggregation execution plan.

[0010] Preferably, the steps for obtaining the partial summary value table are as follows: The city-level node aggregation execution scheme is distributed to multiple independent data source nodes. The underlying stored data columns corresponding to the substation material arrival quantity, the number of construction team members entering the site, and the voltage level group identifier are read from each independent data source node. According to the multi-dimensional aggregation operator in the city-level node aggregation execution scheme, the underlying stored data columns of substation material arrival quantity under the same voltage level group identifier are grouped and summarized, and the underlying stored data columns of construction team members entering the site under the same voltage level group identifier are grouped and summarized to obtain a partial summary value table.

[0011] Preferably, the step of obtaining the shuffle distribution data block is as follows: Read the data entities corresponding to the voltage level group identifiers in the local summary value table line by line, determine the data entities as voltage level group values, read the encoding position of each character according to the character order of the voltage level group values, convert the encoding position into hash encoding, write the hash encoding into the encoding coordinates of the consistent hash routing ring, compare the encoding coordinates and slot node coordinates one by one according to the set direction of the consistent hash routing ring, determine the nearest slot node as the target slot node corresponding to the voltage level group value, and form the slot mapping relationship of the local summary value table; Based on the slot mapping relationship of the local summary value table, read the voltage level grouping value, substation material arrival quantity local summary value, and construction team entry personnel local summary value corresponding to each row of the local summary value table. Retrieve the target slot node in the slot mapping relationship of the local summary value table according to the voltage level grouping value. Write the data row corresponding to the same target slot node into the data shuffling channel of the corresponding slot node. According to the channel number of the data shuffling channel, the node number of the target slot node, and the hash encoding order of the voltage level grouping value, the data row written into the data shuffling channel is divided into blocks and encapsulated to generate shuffling and distribution data blocks.

[0012] Preferably, the steps for obtaining the multi-dimensional project management cursor are as follows: Locally collect the shuffling distribution data blocks in the corresponding data shuffling channel transmission at each slot node, read the channel number, the node number of the target slot node, and the hash code of the voltage level grouping value carried by the shuffling distribution data block, check whether the channel number belongs to the node number of the target slot node, write the shuffling distribution data block that matches the check into the memory buffer of the slot node, read the data row at the current reading position in each data shuffling channel in the encoding order from front to back according to the hash code of the voltage level grouping value, compare the hash code of the voltage level grouping value corresponding to each data row, write the data row with the hash code arranged first into the continuous output position, and establish a memory pipelined aggregation stream; According to the memory pipelined aggregation stream, the values ​​of the substation material arrival quantity column and the construction team entry personnel column associated with the same voltage level group value are read sequentially. The current accumulation state corresponding to the substation material arrival quantity column value and the construction team entry personnel column value are recorded. Before performing the superposition and summation operation, the current accumulation state is combined with the value to be accumulated to form an estimated sum. The estimated sum is compared with the preset overflow prevention threshold condition item by item. If the estimated sum does not pass the preset overflow prevention threshold condition, the storage bit width corresponding to the current accumulation state is expanded and superposition and summation are performed. If the estimated sum passes the preset overflow prevention threshold condition, superposition and summation are performed directly to generate a multi-dimensional aggregated value. According to the fixed field order of voltage level grouping values, substation material arrival quantity summary results, and construction team entry personnel summary results, the multidimensional aggregated values ​​are written to the read-only storage area. The write start position, write end position, and field offset position corresponding to the read-only storage area are locked and structured into a read-only data snapshot. The global virtual address of the read-only data snapshot in the memory buffer is extracted as a dynamic read pointer. The dynamic read pointers generated by all slot nodes are merged according to the node number order of the slot nodes. The merged dynamic read pointers are encapsulated into an index structure. The index structure is assembled into the management memory pool to generate a multidimensional project management cursor.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, focusing on the power project management scenario, the quantity of substation materials arriving, the number of construction team members on site, and voltage level grouping identifiers are simultaneously extracted from project request items. Through lexical decomposition, syntax parsing, data type node traversal, and project code node sorting, a power project query syntax tree is formed. This transforms the request content, originally scattered across fields, codes, and grouping identifiers, into an executable semantic structure with hierarchical relationships. Subsequently, multidimensional cost estimates are calculated by combining the node hierarchy depth of multidimensional aggregation operators and the data cardinality characteristics of voltage level grouping identifiers. This allows aggregation tasks to be pre-scheduled according to the data source node hierarchy, reducing invalid cross-node transmission and lowering redundant aggregation overhead. Furthermore, the multidimensional aggregation operators are decomposed and distributed to the city-level distributed node execution task queues, enabling the quantity of substation materials arriving and the number of construction team members on site to be processed from independent data sources. Local aggregation is completed within nodes, and the locally summarized numerical table then enters the consistent hash routing ring based on the hash encoding of the voltage level group values. This can guide data rows corresponding to the same voltage level group values ​​to nodes in the same slot, improving the consistency of aggregation results across cities. After the shuffled and distributed data blocks are loaded into the memory buffer, they are merged in the encoded order to form a memory pipelined aggregation stream. Before the aggregation is overlaid, the current accumulation status and the estimated sum of the values ​​to be accumulated are verified based on the anti-overflow threshold condition. This allows the numerical aggregation process to perform dynamic upgrades or direct aggregation based on storage bandwidth risk, balancing aggregation accuracy and memory usage control. The multidimensional aggregated numerical structure is converted into a read-only data snapshot, and the dynamically read pointers are merged and encapsulated into an index structure and assembled into the management memory pool. This allows the project management side to directly locate the multidimensional aggregation results in the form of a cursor, improving query response efficiency. Attached Figure Description

[0014] Fig. 1 A schematic diagram illustrating the multidimensional cost prediction values; Fig. 2 This is a partial summary table of values. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figs. 1-2 This invention provides a technical solution: a method for dynamic aggregation of multi-dimensional data for power project management, comprising the following steps: The project request items are composed of the substation material arrival quantity, the number of construction team members entering the site, and the voltage level grouping identifier. An abstract syntactic feature sequence is established through lexical and syntactic analysis. The data type nodes and project coding nodes in the abstract syntactic feature sequence are traversed and sorted to construct hierarchical relationship key-value pairs and generate a power project query syntax tree. By combining the multidimensional aggregation operators and each identifier in the power project query syntax tree, multidimensional cost estimates are calculated to establish push-down aggregation scheduling tasks; multidimensional aggregation operators are decomposed and assigned to the city-level distribution node execution task queue to generate city-level node aggregation execution schemes. The city-level node aggregation execution plan is assigned to independent data source nodes. The underlying stored data columns are read to perform local aggregation and generate a local summary value table. The voltage level grouping values ​​are extracted and mapped to the consistent hash routing ring addressing slot nodes through hash encoding. The data is then distributed through the data shuffling channel to generate shuffling distribution data blocks. Collect shuffled distribution data blocks and load them into a memory buffer. Then, establish a memory pipelined aggregation stream through multi-way merge sorting. Before performing the overlay aggregation operation, verify the estimated sum according to the overflow prevention threshold. If the verification fails, perform dynamic data type upgrade before aggregation. If the verification passes, aggregate directly to generate multidimensional aggregated values. Structure the multidimensional aggregated values ​​into read-only data snapshots and extract the dynamic read pointers. Encapsulate them into an index structure and assemble them into the management memory pool to generate a multidimensional project management cursor.

[0017] The steps for obtaining the abstract grammatical feature sequence are as follows: Extract the quantity of materials delivered to the substation, the number of personnel entering the construction team, and the voltage level grouping identifier. Read the field name, field value, field source location, and project code attribution mark corresponding to each item. Concatenate them according to the arrangement rule of field name first, field value in the middle, and project code attribution mark last. Remove empty fields, duplicate fields, and fields without project code attribution mark to obtain the project request item. Based on field name boundaries, field value boundaries, and project code attribution marker boundaries, the project request item is broken down character by character. The start position, end position, and field attribution category of each character segment are recorded. The substation material arrival quantity, construction team personnel, and voltage level grouping identifier are marked as data retrieval identifiers. The start and end positions of the character segments are organized into character boundary information. The data retrieval identifiers and character boundary information are arranged sequentially according to the field attribution category to establish an abstract grammatical feature sequence.

[0018] Specifically, based on the extracted substation material arrival quantities, construction team personnel, and voltage level grouping identifiers, the system first batches raw data records are read from the interfaces of the power project management system, material resource planning system, and human resource scheduling system. For each record, the corresponding field names are extracted programmatically, such as "material_arrival_quantity", field values, such as "150.5", field source locations, such as "ERP_DB.dbo.table_material@row_id:1024", and project code attribution markers, such as "PJ-2026-HB-01-220KV". Then, following the rule of "field name first, field value centered, project code attribution marker last", and using commas as separators, this information is concatenated into a standardized string, specifically in the form of... The string "material_arrival_quantity,150.5,PJ-2026-HB-01-220KV" is processed using a set of data cleaning rules to filter fields. Specifically, a temporary hash set is created to store the hash values ​​of the generated strings. After each concatenation, the hash value is calculated. If the hash value already exists in the set, it is considered a duplicate field and is removed. The rules also check if the field value is NULL, an empty string, or contains only whitespace characters; if so, it is considered an empty field and is removed. Finally, the regular expression "^PJ-\d{4}-[AZ]{2}-\d{2}-.+$" is used to validate the format of the project code attribution marker. Records that do not conform to this format are considered to lack a valid project code attribution marker and are removed. All cleaned and concatenated strings are then combined to obtain the project request item.

[0019] Based on the project request items generated in the previous step, a lexical analysis program needs to be started to process each item in order to structure them into the form required for subsequent syntax tree construction. This program first segments each project request item string using a preset comma separator as the boundary, breaking it down into three independent character segments: field name, field value, and project code attribution marker. During the segmentation process, the program simultaneously records the precise position of each segment in the original string; specifically, it records the starting and ending character indices of each character segment. For example, for the string "material_arrival_quantity,150.5,PJ-2026-HB-01-220KV", the first segment "material_arrival_quantity" starts at position 0 and ends at position 24, and the second segment "150.5"... The starting position is 26, and the ending position is 30. The starting position of the third segment, "PJ-2026-HB-01-220KV", is 32, and the ending position is 53. Subsequently, for each field name, such as the quantity of materials delivered to the substation, the number of construction team members entering the site, and the voltage level grouping identifier, a unique integer ID is queried from the internal metadata dictionary and assigned as its data retrieval identifier. The starting and ending positions of the recorded character segments are encapsulated into a character boundary information structure containing two integer values. Finally, the entries are arranged strictly in the order of "field category, data retrieval identifier, and character boundary information" to form a list containing multiple structured entries. The field category is marked as "NAME", "VALUE", and "PROJECT_ID" according to the source of the segment (parts one, two, and three), respectively. Finally, this list is solidified as a whole to establish an abstract syntactic feature sequence.

[0020] The steps to obtain the query syntax tree for power projects are as follows: Based on the abstract grammatical feature sequence, the data type nodes corresponding to the substation material arrival quantity, the data type nodes corresponding to the number of construction team members, and the data type nodes corresponding to the voltage level group identifier are retrieved. The project code nodes corresponding to each data type node are then retrieved. The traversal and sorting are performed in the hierarchical order of root node, data type node, and project code node. The parent pointer, child pointer, and field mapping relationship between adjacent nodes are recorded. The parent pointer, child pointer, and field mapping relationship are extracted as node association edge attributes. The node association edge attributes are constructed into hierarchical relationship key-value pairs according to the hierarchical number and field category to generate a power project query syntax tree.

[0021] Specifically, based on the abstract syntactic feature sequence, to construct a query structure that reflects the hierarchy and logical relationships between data, an empty root node is first initialized as the starting point of the power project query syntax tree. Then, each structured entry in the abstract syntactic feature sequence is traversed. When the entry's field belongs to the category "NAME", a corresponding data type node is searched and instantiated in a predefined node template library based on its data retrieval identifier, such as "substation material arrival quantity". This template library predefines node structures for basic data types such as "integer", "floating-point", and "string", and uses them as direct children of the root node. Then, the "PROJECT_ID" entries associated with this "NAME" entry in the sequence are processed. Similarly, a project code node is instantiated based on its content and attached to the newly created data type node, thus forming a three-level hierarchical structure of "root node - data type node - project code node". Each time a node is attached, the association information between nodes is recorded synchronously. Specifically, a parent pointer attribute is set in the child node object to store the memory address or unique ID of its parent node. At the same time, a child pointer list is maintained in the parent node object to add the address or ID of the newly added child node. The field mapping relationship between the data type node and the project code node, i.e., the specific project code string, is stored as the attribute connecting this edge. After traversal, all nodes and their associated edge attributes are organized in a unified manner according to the format "{'node_id': {'parent_id': ..., 'child_ids':[...], 'edge_properties': {...}}}". The hierarchy number is automatically generated based on the node depth, and the field category is part of the edge attribute. Finally, a power project query syntax tree represented by hierarchical relationship key-value pairs is constructed.

[0022] The steps to obtain the pushdown aggregation scheduling task are as follows: Extract multidimensional aggregation operators, substation material arrival quantities, construction team personnel, and voltage level grouping identifiers from the power project query syntax tree. Read the parent node number, child node number, and hierarchy depth marker of the node containing the multidimensional aggregation operator. Count the number of different grouping values ​​corresponding to the voltage level grouping identifier under the project coding node. Compare the hierarchy depth marker and the number of different grouping values ​​item by item. Calculate the multidimensional cost prediction value of the data source node hierarchy according to the cost accumulation order of the data source node hierarchy. Establish a push-down aggregation scheduling task.

[0023] Specifically, using the constructed power project query syntax tree, to optimize subsequent aggregation calculations, the computational cost of different aggregation paths needs to be pre-assessed. First, a depth-first traversal is performed starting from the root node of the syntax tree to locate all nodes marked as multi-dimensional aggregation operators, such as operation nodes marked "SUM" or "COUNT". The query dimensions associated with these nodes are extracted, namely the quantity of materials arriving at the substation, the number of personnel entering the construction team, and the grouping basis, namely the voltage level grouping identifier. Simultaneously, the parent node number, child node number, and its level depth in the tree are read from the aggregation operator node (e.g., a level depth of 3). Then, the program drills down to the project code nodes associated with the voltage level grouping identifier. By deduplicating and counting the field values ​​under these nodes, the specific number of values ​​for different groups is obtained. For example, if "500" exists... Given three different voltage levels, "220kV", "110kV", and "220kV", the number of different group values ​​is 3. Then, the hierarchical depth marker of the aggregation operator node is compared with this number of group values. Combined with the pre-configured hierarchical affiliation of data source nodes (for example, some data source nodes are physically located in city A, and others are located in city B), the multi-dimensional cost estimate value for performing aggregation operations at different data source node levels is calculated according to the cost accumulation order, such as accumulating network transmission costs first, and then accumulating node computation costs. This value calculation comprehensively considers data transmission volume, CPU computation cycle and memory usage. Finally, each possible aggregation pushdown path and its corresponding cost estimate value are recorded to establish a pushdown aggregation scheduling task list containing task ID, target node, operation type and estimated cost value.

[0024] The steps to obtain the city-level node aggregation execution plan are as follows: Read the multidimensional cost estimation value, data source node hierarchical relationship and cost accumulation order in the push-down aggregation scheduling task, determine the push-down priority of the multidimensional aggregation operator according to the multidimensional cost estimation value in ascending order, decompose the multidimensional aggregation operator into sub-tasks of substation material arrival quantity aggregation, construction team entry personnel aggregation, and voltage level grouping identifier grouping, and allocate them to the corresponding city distribution nodes according to the data source node hierarchical relationship, and obtain the city distribution node execution task queue; Based on the task queue of the city-level distribution nodes, read the sub-tasks for aggregating substation material arrivals, construction team personnel, and voltage level grouping identifiers corresponding to each city-level distribution node. Verify the node execution order, field value range, and project code attribution flag for each sub-task. Write the decomposed multidimensional aggregation operator, substation material arrivals, construction team personnel, and voltage level grouping identifier into the task slots of the corresponding city-level distribution nodes to form a fixed mapping relationship between node task slots and field attribution categories, and generate the city-level node aggregation execution plan.

[0025] Specifically, based on the multi-dimensional cost estimates calculated for each potential execution path in the pushdown aggregation scheduling task, to generate the optimal distributed execution plan, all entries in the task list are first sorted in ascending order according to their multi-dimensional cost estimates. This is done to prioritize the aggregation pushdown scheme with the lowest computational and network overhead. Then, the scheme at the top of the sorted list, i.e., the one with the lowest cost, is selected as the final pushdown priority. Next, the macro-level multi-dimensional aggregation operators defined in this scheme are logically decomposed into a series of finer-grained atomic-level subtasks. Specifically, this involves subtasks related to the quantity of materials arriving at the substation. The summation operation is broken down into a subtask for aggregating the amount of materials arriving at the substation, a subtask for aggregating the number of construction team members entering the site, and a subtask for aggregating the number of construction team members entering the site. After the voltage level grouping operation is broken down into a subtask for aggregating voltage level grouping identifiers, these subtasks are precisely assigned to the corresponding city-level distribution nodes based on the hierarchical relationship of the data source nodes recorded in the push-down aggregation scheduling task. For example, the material data aggregation task for city A is assigned to the computing node for city A, and the personnel data aggregation task for city B is assigned to the computing node for city B, thus forming a node execution task queue distributed by city.

[0026] Based on the task queue generated in the previous step, to ensure that each distributed node can correctly execute its assigned tasks, a detailed execution plan needs to be generated for each node. First, the task queue is traversed, visiting each city-level distributed node in turn, and reading the sub-tasks containing sub-tasks such as substation material arrival quantity aggregation, construction team personnel aggregation, and voltage level grouping identifier grouping. Then, each sub-task undergoes rigorous parameter verification, including checking whether its specified node execution order number matches the current node, and verifying whether the range of values ​​for the fields it processes falls within the jurisdiction of that city-level node. For example, city-level node A only processes projects with "-C" in their code. The data marked with "D-" and the validity of the project code attribution mark are verified. After verification, the disassembled multidimensional aggregation operators, specific metrics (such as the quantity of materials delivered to the substation and the number of construction team members on site), and grouping criteria (such as voltage level grouping identifiers) are written into the pre-defined task slots within the corresponding city-level distributed nodes. These task slots are fixed areas in memory. Each slot is bound to a specific field attribution category (such as metric or dimension) through its offset address. In this way, a fixed mapping relationship between node task slots and field attribution categories is established within each distributed node. Finally, the slot configuration information of all nodes is collected to generate the city-level node aggregation execution plan.

[0027] The steps to obtain the partial summary value table are as follows: The city-level node aggregation execution plan is distributed to multiple independent data source nodes. The underlying stored data columns corresponding to the substation material arrival quantity, the number of construction team members entering the site, and the voltage level group identifier are read from each independent data source node. According to the multi-dimensional aggregation operator in the city-level node aggregation execution plan, the underlying stored data columns of substation material arrival quantity under the same voltage level group identifier are grouped and summarized, and the underlying stored data columns of construction team members entering the site under the same voltage level group identifier are grouped and summarized to obtain a partial summary value table.

[0028] Specifically, after the city-level node aggregation execution plan is distributed, each independent distributed data source node immediately begins executing its local computation task. Each node first parses the received execution plan portion, identifying the data column identifiers that need to be processed, namely, the quantity of substation materials arriving, the number of construction team members on site, and the voltage level group identifier. Then, the node program directly accesses its local underlying storage system, which can be a columnar database or a distributed file system, and reads the complete data column corresponding to the quantity of substation materials arriving into memory based on the identifier. Similarly, it reads the underlying storage data column corresponding to the number of construction team members on site, as well as the underlying storage data column corresponding to the voltage level group identifier. After the data is loaded, the node begins local aggregation based on the multi-dimensional aggregation operators specified in the execution plan. The aggregation operation is executed as follows: the program uses the voltage level group identifier column as the driver to group the data. For each unique voltage level group identifier, such as "220kV", the program will filter out all the material arrival data corresponding to "220kV" substations and perform a summation operation on these data to obtain a local summary value of material arrival. At the same time, it will filter out all the construction team personnel data corresponding to "220kV" and perform a summation operation on these data to obtain a local summary value of personnel entry. This process is repeated for all unique voltage level group identifiers. Finally, each voltage level group identifier and its two corresponding local summary values ​​are combined into a record. All records together form a local summary value table in memory.

[0029] The steps for obtaining the shuffle distribution data blocks are as follows: Read the data entities corresponding to the voltage level group identifiers in the local summary value table line by line, determine the data entities as voltage level group values, read the encoding position of each character in the character order of the voltage level group values, convert the encoding position into hash code, write the hash code into the encoding coordinates of the consistent hash routing ring, compare the encoding coordinates and slot node coordinates one by one according to the set direction of the consistent hash routing ring, determine the nearest slot node as the target slot node corresponding to the voltage level group value, and form the slot mapping relationship of the local summary value table; Based on the slot mapping relationship of the local summary value table, read the voltage level grouping value, substation material arrival quantity local summary value, and construction team entry personnel local summary value corresponding to each row in the local summary value table. Retrieve the target slot node in the slot mapping relationship of the local summary value table according to the voltage level grouping value. Write the data row corresponding to the same target slot node into the data shuffling channel of the corresponding slot node. According to the channel number of the data shuffling channel, the node number of the target slot node, and the hash encoding order of the voltage level grouping value, the data row written into the data shuffling channel is divided into blocks and encapsulated to generate shuffling and distribution data blocks.

[0030] Specifically, after each independent data source node generates a local summary value table, in order to perform global aggregation, data with the same voltage level grouping value needs to be redistributed to the same target node. First, the local summary value table is scanned row by row to extract the data entity corresponding to the voltage level grouping identifier in each row and identify it as the voltage level grouping value, such as "110kV". Then, a hash function, such as MurmurHash3, is applied to the string of the voltage level grouping value itself. This function calculates a 32-bit or 64-bit hash code based on the character sequence and its encoding position. This hash code is then mapped to a pre-built consistent hash routing ring. Logically, the ring is a closed loop, on which coordinates representing all computation slot nodes are distributed. These node coordinates are also obtained by hashing their unique identifiers (such as IP addresses plus port numbers). The mapping operation is to place the hash code of the voltage level group value as a new coordinate point on the ring. Then, starting from this coordinate point, the loop moves in a preset clockwise or counterclockwise direction. The coordinates of the first slot node encountered are determined as the target slot node corresponding to the voltage level group value. By repeating this process for each row of the local summary value table, a complete mapping relationship from each row of the table to the specific target slot node is finally formed, namely the local summary value table slot mapping relationship.

[0031] Based on the slot mapping relationship established in the previous step for the local summary value table, the data shuffling process is initiated. The program first iterates through each row of data in the local summary value table, reading the voltage level grouping value, the local summary value of substation material arrival quantity, and the local summary value of the number of construction team members present. For each row, the program uses its voltage level grouping value as the key to query the slot mapping relationship in the local summary value table, quickly retrieving the target slot node to which the data should be sent. Then, data rows with the same target slot node are logically grouped together, and these data rows are written one by one into a dedicated network transmission channel connected to that target slot node, i.e., the data shuffling channel. To improve transmission efficiency... For efficient transmission and easy management, the data written to the channel is not sent in a scattered manner, but is first accumulated in the buffer at the sending end. When the accumulated data volume reaches a preset threshold (e.g., 1MB) or the waiting time exceeds a preset duration (e.g., 100 milliseconds), it will be encapsulated in blocks. During encapsulation, a header information is added to each data block. This header information includes the unique channel number of the data shuffling channel, the unique node number of the target slot node, and the hash code range of the voltage level grouping value corresponding to the data in the block. The data blocks are organized in this order of "channel number-node number-hash code" to finally generate a series of shuffling and distribution data blocks ready to be sent to different target slot nodes.

[0032] The steps to obtain the multi-dimensional project management cursor are as follows: Locally collect shuffling distribution data blocks from the corresponding data shuffling channel transmission at each slot node. Read the channel number, the node number of the target slot node, and the hash code of the voltage level grouping value carried in the shuffling distribution data block. Verify whether the channel number belongs to the node number of the target slot node. Write the shuffling distribution data blocks that match the verification to the memory buffer of the slot node. Read the data rows at the current reading position in each data shuffling channel one by one according to the hash code of the voltage level grouping value from front to back. Compare the hash codes of the voltage level grouping values ​​corresponding to each data row. Write the data rows with the hash codes arranged first to the continuous output position to establish a memory pipelined aggregation stream. Based on the memory pipelined aggregation stream, the values ​​of the substation material arrival quantity column and the construction team entry personnel column associated with the same voltage level group value are read sequentially. The current accumulation status corresponding to the substation material arrival quantity column value and the construction team entry personnel column value is recorded. Before performing the overlay and summary operation, the current accumulation status is combined with the value to be accumulated to form an estimated sum. The estimated sum is compared with the preset overflow prevention threshold condition item by item. If the estimated sum does not pass the preset overflow prevention threshold condition, the storage bit width corresponding to the current accumulation status is expanded and then the overlay and summary is performed. If the estimated sum passes the preset overflow prevention threshold condition, the overlay and summary is performed directly to generate a multi-dimensional aggregated value. Following the fixed field order of voltage level grouping values, substation material arrival quantity summary results, and construction team entry personnel summary results, the multidimensional aggregated values ​​are written to the read-only storage area. The write start position, write end position, and field offset position corresponding to the read-only storage area are locked and structured into a read-only data snapshot. The global virtual address of the read-only data snapshot in the memory buffer is extracted as a dynamic read pointer. The dynamic read pointers generated by all slot nodes are merged according to the node number order of the slot nodes. The merged dynamic read pointers are encapsulated into an index structure. The index structure is assembled into the management memory pool to generate a multidimensional project management cursor.

[0033] Specifically, after the shuffling phase, each slot node begins processing the data received from the data shuffling channel. The receiving program on the node continuously listens to the network port, collecting shuffling and distribution data blocks in transit. Whenever a complete data block arrives, its header information is immediately read, extracting the channel number, the target slot node's node number, and the hash code of the voltage level grouping value. A verification operation is then performed, comparing the target slot node number in the data block with the actual number of the current node. Only if the two match completely is the data block considered valid and accepted. Data blocks that pass the verification are then written into the slot node's dedicated memory. In the memory buffer, once all relevant data blocks have been collected, a multi-way merge sort process is initiated. This process treats each data shuffling channel from which the received data block originates as an independent, hash-sorted input stream. In each iteration, it simultaneously probes the current row (i.e., the next data row to be read) of all input streams, compares the hash codes of the voltage level grouping values ​​corresponding to these rows, selects the row with the smallest hash code, writes it to a contiguous memory output area, and advances the pointer of that input stream by one row. This process is repeated until all input streams are exhausted, thereby establishing a globally ordered memory-piped aggregate stream.

[0034] Based on the established memory-piped aggregation stream, the final global aggregation operation begins. The program sequentially reads data rows from the aggregation stream. Since the data is sorted by voltage level grouping values ​​(hash codes), data rows with the same voltage level grouping values ​​will appear consecutively. The program maintains a current accumulated state variable for each of the two metric columns: substation material arrival quantity and construction team personnel. For example, a 64-bit long integer is used. When a new data row is read, an overflow prevention pre-check is performed before the actual addition operation. This check adds the value of the current accumulated state variable to the value to be accumulated (i.e., the local summary value in the current data row) to form an estimated sum. This estimated sum is then compared with a preset overflow prevention threshold. This threshold is set according to the data type of the current accumulated state variable. For example, for a 64-bit long integer, its maximum value is 9,223,372,036,854,775,807, and the threshold can be set to 98% of this maximum value, which is approximately 9.04 x If the estimated sum exceeds 10^18, it indicates that continuing to use the current data type will cause an overflow. At this time, the program will dynamically upgrade the storage type of the current accumulated state variable from a 64-bit long integer to a data type that supports higher precision and range, such as a high-precision decimal type, and then perform the summation. If the estimated sum does not exceed the threshold, the summation will be performed directly. This operation will be repeated for all data rows with the same voltage level grouping values ​​to finally generate accurate multidimensional aggregated values.

[0035] After generating the final multidimensional aggregated values ​​corresponding to each voltage level group, these results need to be structured and indexed for efficient querying by upper-layer applications. First, following a predetermined and fixed field order of "voltage level group value, substation material arrival summary result, and construction team entry personnel summary result," the calculated multidimensional aggregated values ​​are organized into a compact data structure and continuously written to a read-only memory area. After writing, the system locks the start and end virtual addresses of the memory block occupied by this data structure, as well as the offset of each field relative to the start address, and stores this data in a read-only storage area. These metadata are solidified together to form an immutable read-only data snapshot. Then, the starting virtual address of this read-only data snapshot in the global memory address space is extracted and used as a dynamic read pointer that can directly access this data. All slot nodes report their generated dynamic read pointers to a central coordinator. The coordinator sorts and merges these pointers according to the node number of the slot node, and encapsulates the merged pointer list into an index structure that supports fast lookup, such as a hash table or a B+ tree. Finally, this complete index structure is assembled into a globally shared management memory pool and exposed to the outside world as a unified multi-dimensional project management cursor that can be accessed iteratively by upper-layer applications.

[0036] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for dynamic aggregation of multi-dimensional data for power project management, characterized in that, Includes the following steps: The project request items are composed of the quantity of materials delivered to the substation, the number of personnel entering the construction team, and the voltage level grouping identifier. An abstract grammatical feature sequence is then established through lexical and grammatical analysis. The data type nodes and project code nodes in the abstract syntax feature sequence are traversed and sorted to construct hierarchical key-value pairs and generate a power project query syntax tree. Combining the multidimensional aggregation operators and each identifier in the power project query syntax tree, multidimensional cost estimates are calculated to establish push-down aggregation scheduling tasks; the multidimensional aggregation operators are decomposed and allocated to the city-level distribution node execution task queue to generate a city-level node aggregation execution scheme. The city-level node aggregation execution scheme is assigned to an independent data source node. The underlying stored data columns are read to perform local aggregation and generate a local summary value table. The voltage level grouping values ​​are extracted and mapped to the consistent hash routing ring addressing slot node through hash encoding. The data is then distributed through the data shuffling channel to generate a shuffling distribution data block. The collected shuffled distribution data blocks are loaded into a memory buffer and then a memory pipelined aggregation stream is established through multi-way merge sorting. Before performing the overlay and aggregation operation, the estimated sum is checked according to the overflow prevention threshold condition. If the check fails, the data type is dynamically upgraded before aggregation. If the check passes, the sum is directly aggregated to generate multidimensional aggregated values. The multidimensional aggregated numerical data is structured into a read-only data snapshot and the dynamic read pointer is extracted, encapsulated into an index structure and assembled into the management memory pool to generate a multidimensional project management cursor.

2. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the abstract grammatical feature sequence are as follows: Extract the quantity of materials delivered to the substation, the number of personnel entering the construction team, and the voltage level grouping identifier. Read the field name, field value, field source location, and project code attribution mark corresponding to each item. Concatenate them according to the arrangement rule of field name first, field value in the middle, and project code attribution mark last. Remove empty fields, duplicate fields, and fields without project code attribution mark to obtain the project request item. The project request item is decomposed character by character according to the field name boundary, field value boundary, and project code attribution mark boundary. The start position, end position, and field attribution category of each character segment are recorded. The substation material arrival quantity, construction team personnel, and voltage level grouping identifier are marked as data retrieval identifiers. The start and end positions of the character segments are organized into character boundary information. The data retrieval identifiers and character boundary information are arranged in order according to the field attribution category to establish an abstract grammatical feature sequence.

3. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the query syntax tree for the power project are as follows: Based on the abstract grammatical feature sequence, retrieve the data type nodes corresponding to the substation material arrival quantity, the data type nodes corresponding to the number of construction team members entering the site, and the data type nodes corresponding to the voltage level group identifier. Continue to retrieve the project code nodes corresponding to each data type node. Perform traversal sorting according to the hierarchical order of root node, data type node, and project code node. Record the parent pointer, child pointer, and field mapping relationship between adjacent nodes. Extract the parent pointer, child pointer, and field mapping relationship as node association edge attributes. Construct the node association edge attributes into hierarchical relationship key-value pairs according to the hierarchical number and field belonging category to generate a power project query syntax tree.

4. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the push-down aggregation scheduling task are as follows: Extract the multidimensional aggregation operator, substation material arrival quantity, construction team personnel, and voltage level grouping identifier from the power project query syntax tree. Read the parent node number, child node number, and hierarchy depth marker of the node where the multidimensional aggregation operator is located. Count the number of different grouping values ​​corresponding to the voltage level grouping identifier under the project coding node. Compare the hierarchy depth marker and the number of different grouping values ​​item by item. Calculate the multidimensional cost prediction value of the data source node hierarchy according to the cost accumulation order of the data source node hierarchy. Establish a push-down aggregation scheduling task.

5. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the city-level node aggregation execution plan are as follows: Read the multidimensional cost estimation value, data source node hierarchical relationship and cost accumulation order in the push-down aggregation scheduling task, determine the push-down priority of the multidimensional aggregation operator according to the multidimensional cost estimation value in ascending order, decompose the multidimensional aggregation operator into sub-tasks of substation material arrival quantity aggregation, construction team entry personnel aggregation, and voltage level group identification grouping, and allocate them to the corresponding city distribution nodes according to the data source node hierarchical relationship, and obtain the city distribution node execution task queue; Based on the task queue of the city-level distribution nodes, read the sub-tasks for aggregating substation material arrivals, construction team personnel, and voltage level grouping identifiers corresponding to each city-level distribution node. Verify the node execution order, field value range, and project code attribution flag for each sub-task. Write the decomposed multi-dimensional aggregation operator, substation material arrivals, construction team personnel, and voltage level grouping identifier into the task slots of the corresponding city-level distribution nodes to form a fixed mapping relationship between node task slots and field attribution categories, and generate a city-level node aggregation execution plan.

6. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the local summary value table are as follows: The city-level node aggregation execution scheme is distributed to multiple independent data source nodes. The underlying stored data columns corresponding to the substation material arrival quantity, the number of construction team members entering the site, and the voltage level group identifier are read from each independent data source node. According to the multi-dimensional aggregation operator in the city-level node aggregation execution scheme, the underlying stored data columns of substation material arrival quantity under the same voltage level group identifier are grouped and summarized, and the underlying stored data columns of construction team members entering the site under the same voltage level group identifier are grouped and summarized to obtain a partial summary value table.

7. The method for dynamic aggregation of multi-dimensional data for power project management according to claim 1, characterized in that, The steps for obtaining the shuffle distribution data block are as follows: Read the data entities corresponding to the voltage level group identifiers in the local summary value table line by line, determine the data entities as voltage level group values, read the encoding position of each character according to the character order of the voltage level group values, convert the encoding position into hash encoding, write the hash encoding into the encoding coordinates of the consistent hash routing ring, compare the encoding coordinates and slot node coordinates one by one according to the set direction of the consistent hash routing ring, determine the nearest slot node as the target slot node corresponding to the voltage level group value, and form the slot mapping relationship of the local summary value table; Based on the slot mapping relationship of the local summary value table, read the voltage level grouping value, substation material arrival quantity local summary value, and construction team entry personnel local summary value corresponding to each row of the local summary value table. Retrieve the target slot node in the slot mapping relationship of the local summary value table according to the voltage level grouping value. Write the data row corresponding to the same target slot node into the data shuffling channel of the corresponding slot node. According to the channel number of the data shuffling channel, the node number of the target slot node, and the hash encoding order of the voltage level grouping value, the data row written into the data shuffling channel is divided into blocks and encapsulated to generate shuffling and distribution data blocks.

8. The method for dynamic aggregation of multi-dimensional data for power project management and control according to claim 1, characterized in that, The steps for obtaining the multi-dimensional project management cursor are as follows: Locally collect the shuffling distribution data blocks in the corresponding data shuffling channel transmission at each slot node, read the channel number, the node number of the target slot node, and the hash code of the voltage level grouping value carried by the shuffling distribution data block, check whether the channel number belongs to the node number of the target slot node, write the shuffling distribution data block that matches the check into the memory buffer of the slot node, read the data row at the current reading position in each data shuffling channel in the encoding order from front to back according to the hash code of the voltage level grouping value, compare the hash code of the voltage level grouping value corresponding to each data row, write the data row with the hash code arranged first into the continuous output position, and establish a memory pipelined aggregation stream; According to the memory pipelined aggregation stream, the values ​​of the substation material arrival quantity column and the construction team entry personnel column associated with the same voltage level group value are read sequentially. The current accumulation state corresponding to the substation material arrival quantity column value and the construction team entry personnel column value are recorded. Before performing the superposition and summation operation, the current accumulation state is combined with the value to be accumulated to form an estimated sum. The estimated sum is compared with the preset overflow prevention threshold condition item by item. If the estimated sum does not pass the preset overflow prevention threshold condition, the storage bit width corresponding to the current accumulation state is expanded and superposition and summation are performed. If the estimated sum passes the preset overflow prevention threshold condition, superposition and summation are performed directly to generate a multi-dimensional aggregated value. According to the fixed field order of voltage level grouping values, substation material arrival quantity summary results, and construction team entry personnel summary results, the multidimensional aggregated values ​​are written to the read-only storage area. The write start position, write end position, and field offset position corresponding to the read-only storage area are locked and structured into a read-only data snapshot. The global virtual address of the read-only data snapshot in the memory buffer is extracted as a dynamic read pointer. The dynamic read pointers generated by all slot nodes are merged according to the node number order of the slot nodes. The merged dynamic read pointers are encapsulated into an index structure. The index structure is assembled into the management memory pool to generate a multidimensional project management cursor.

Images