A distributed memory data management method, device and storage medium for a power grid infrastructure project

Abstract

The application provides a distributed memory data management method and device for power grid infrastructure projects and a storage medium. The method comprises the following steps: saving power grid infrastructure multi-source data collected by a collection node in a memory node directly connected to the collection node after cleaning, and using a cold and hot data hierarchical memory scheduling algorithm to complete cold and hot data classification and memory layered storage in all memory nodes; using a user terminal to receive an operation request of a user, using a scheduling node to locate a memory node corresponding to the operation request based on a decentralized consistency protocol based on the type of the operation request; judging whether the operation corresponding to the operation request is a cross-memory node operation, and if so, triggering a cross-node data interaction algorithm based on remote direct memory access to prepare cross-node data interaction; and feeding back a processing result to the user. The application solves the consistency, performance and reliability of distributed memory data management in the business scenario of power grid infrastructure.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and power grid infrastructure engineering integration technology, specifically to a distributed memory data management method, device and storage medium for power grid infrastructure projects. Background Technology

[0002] Power grid infrastructure projects involve construction progress, safety monitoring, equipment arrival, as-built documentation, audit reports, historical acceptance data, etc. The real-time requirements of various data vary, and the existing data scheduling algorithms are not integrated with business scenarios, nor do they predict peak data usage based on operational data.

[0003] Traditional centralized consensus protocols (such as Raft) suffer from single-point-of-failure risks, high concurrency write latency, and overload of coordination nodes in multi-node clusters of power grid infrastructure, meaning they cannot dynamically adjust node roles according to business scenarios.

[0004] Traditional memory scheduling algorithms (such as simple LRU) do not take into account the characteristics of network infrastructure services, which can easily lead to hot data being replaced and memory resources being wasted.

[0005] Power grid infrastructure data has strong correlation characteristics (such as BIM model data and equipment ledgers, construction progress data need to be queried across nodes). Cross-node data transmission under the traditional TCP / IP protocol has problems such as high CPU usage, large transmission latency and low bandwidth utilization, which seriously affects the efficiency of multi-node parallel computing. Summary of the Invention

[0006] In view of one or more technical defects in the prior art, the present invention proposes the following technical solution.

[0007] A distributed memory data management method for power grid infrastructure projects, the method comprising: The storage step involves cleaning the multi-source power grid infrastructure data collected by the acquisition nodes and storing it in memory nodes directly connected to the acquisition nodes. A cold and hot data hierarchical memory scheduling algorithm is used to classify and store cold and hot data in all memory nodes in a hierarchical memory manner. Request steps: Use a user terminal to receive the user's operation request, and use a scheduling node to locate the memory node corresponding to the operation request based on the type of the operation request and a decentralized consensus protocol; The judgment step is to determine whether the operation corresponding to the operation request is a cross-memory node operation. If so, the cross-node data interaction algorithm based on remote direct memory access is triggered to prepare for cross-node data interaction. The feedback step involves performing the corresponding operation based on the type of the operation request and then feeding back the processing result to the user.

[0008] Furthermore, the hot and cold data hierarchical memory scheduling algorithm is as follows: a hot and cold data identification value is constructed based on three dimensions: business weight, access frequency, and access time. Data classification is achieved through quantitative scoring. All memory nodes are divided into three levels according to read and write speed: high-speed memory, medium-speed memory, and low-speed cache, corresponding to the storage of hot data, warm data, and cold data, respectively. Differentiated storage and scheduling of data with different priorities are realized. Based on the historical data access time sequence characteristics of power grid infrastructure business, a business-aware memory preloading and replacement strategy is constructed.

[0009] Furthermore, the Hot-Score calculation formula for the hot / cold data identification value is as follows: Hot-Score =α×F-Score +β×B-Score +γ×T-Score; Where α+β+γ=1, the weights are dynamically adjusted according to the business scenario, with a default α=0.4, β=0.3, and γ=0.3; a scoring threshold is set: Hot-Score ≥ 0.7 is considered hot data, and 0.3 ≤ Hot-Score 0.7 indicates warm data, Hot-Score 0.3 is considered cold data; F-Score is an access frequency score, which counts the number of times a data shard is accessed in the past 7 days and is divided into 5 levels according to the percentile. The higher the level, the higher the score. B-Score is a business weight score, which is assigned based on the priority of power grid infrastructure business. Progress management and safety management data are assigned a value of 0.9~1.0, quality management data is assigned a value of 0.7~0.9, cost management and material management data is assigned a value of 0.5~0.7, and historical archived data is assigned a value of 0.1~0.5. T-Score is the access time score, which is the interval between the last access to the data and the current time. The shorter the interval, the higher the score. The calculation formula is: T-Score = e^(-Δt / λ), where Δt is the time interval and λ is the decay coefficient, which is 7 days.

[0010] Furthermore, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak data operation period and high-frequency access data corresponding to the business scenario. A scheduled task is set to load the high-frequency access data into the high-speed memory before the peak data operation period arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The cold and hot data identification values ​​of the data are updated in real time using monitoring nodes. When the access frequency of cold data suddenly increases, the data upgrade process is triggered to adjust the cold data into warm data and load it into the medium-speed memory.

[0011] This ensures business continuity, reduces access latency, and prevents core business data from being replaced.

[0012] Furthermore, the decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. The roles of memory nodes include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, automatically incremented by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure the uniqueness of the time series; and PrevVer is the predecessor version number of the current version. A version chain of data modifications is constructed to achieve historical data traceability.

[0013] Furthermore, the method for dynamically adjusting the role of memory nodes based on their performance metrics and business data weights is as follows: Calculate the Role-Score of a memory node: ; Among them, Pcpu represents CPU availability, Pmem represents memory availability, PnetW represents network bandwidth availability, B-Score is the service data weight score, and a Role-Score greater than or equal to the first threshold is used as a monitoring node, a Role-Score less than the second threshold is used as a data node, and the rest are used as scheduling nodes.

[0014] Furthermore, the operation request type is either a read operation or a write operation. For a read operation, if reading the latest version of data, the user terminal uses the latest version data from its local memory node. If reading historical version data, the user terminal requests the fragment location of the historical version data from the scheduling node based on the version vector, and obtains the historical version data based on the fragment location. For a write operation, before initiating a write request, the user terminal first obtains the version vector of the current data fragment, generates a new version number, completes the data modification locally, and asynchronously sends a write request + new version vector to the associated scheduling node. After receiving the request, the scheduling node verifies the legality of the modification through a majority voting mechanism, and simultaneously compares the version vectors to determine if there is a conflict. If there is no conflict, consistency confirmation is completed.

[0015] Furthermore, the cross-node data interaction algorithm for Remote Direct Memory Access (RDMA) is as follows: A topology-aware data fragmentation routing strategy is constructed. Monitoring nodes periodically collect cluster network topology data, including the rack to which the memory node belongs, switch connection relationships, and link bandwidth utilization, to construct a topology matrix. Based on this matrix, nodes within the same rack are prioritized as data interaction targets, followed by nodes in different racks within the same switch, and finally nodes across switches. When link bandwidth is overloaded or node load is too high, the algorithm automatically switches to a backup routing path. A zero-copy batch data transmission mechanism is implemented, where memory nodes pre-register their local memory regions and map the data fragments to be transmitted as physical memory pages accessible by RDMA, avoiding data copying between kernel and user modes. For large volumes of data in power grid infrastructure, a block-based batch transmission strategy is adopted, dividing the data into fixed-size blocks and sending them in batches via a reliable RDMA connection, reducing the number of handshakes. After receiving the data block, the receiving memory node verifies data integrity using a CRC32 checksum algorithm. If data loss or errors occur, only the abnormal data block is retransmitted, rather than all data, improving transmission efficiency.

[0016] This invention also proposes a distributed memory data management device for power grid infrastructure projects, the device comprising: The storage unit cleans the multi-source power grid infrastructure data collected by the acquisition node and stores it in the memory node directly connected to the acquisition node. It also uses a hot and cold data hierarchical memory scheduling algorithm to classify and store hot and cold data in all memory nodes in a hierarchical memory manner. Request Unit: Receives user operation requests using a user terminal, and locates the memory node corresponding to the operation request using a scheduling node based on a decentralized consensus protocol, according to the type of the operation request; The judgment unit determines whether the operation corresponding to the operation request is a cross-memory node operation. If so, it triggers a cross-node data interaction algorithm based on remote direct memory access to prepare for cross-node data interaction. The feedback unit performs the corresponding operation based on the type of the operation request and feeds back the processing result to the user.

[0017] Furthermore, the hot and cold data hierarchical memory scheduling algorithm is as follows: a hot and cold data identification value is constructed based on three dimensions: business weight, access frequency, and access time. Data classification is achieved through quantitative scoring. All memory nodes are divided into three levels according to read and write speed: high-speed memory, medium-speed memory, and low-speed cache, corresponding to the storage of hot data, warm data, and cold data, respectively. Differentiated storage and scheduling of data with different priorities are realized. Based on the historical data access time sequence characteristics of power grid infrastructure business, a business-aware memory preloading and replacement strategy is constructed.

[0018] Furthermore, the Hot-Score calculation formula for the hot / cold data identification value is as follows: Hot-Score =α×F-Score +β×B-Score +γ×T-Score; Where α+β+γ=1, the weights are dynamically adjusted according to the business scenario, with a default α=0.4, β=0.3, and γ=0.3; a scoring threshold is set: Hot-Score ≥ 0.7 is considered hot data, and 0.3 ≤ Hot-Score 0.7 indicates warm data, Hot-Score 0.3 is considered cold data; F-Score is an access frequency score, which counts the number of times a data shard is accessed in the past 7 days and is divided into 5 levels according to the percentile. The higher the level, the higher the score. B-Score is a business weight score, which is assigned based on the priority of power grid infrastructure business. Progress management and safety management data are assigned a value of 0.9~1.0, quality management data is assigned a value of 0.7~0.9, cost management and material management data is assigned a value of 0.5~0.7, and historical archived data is assigned a value of 0.1~0.5. T-Score is the access time score, which is the interval between the last access to the data and the current time. The shorter the interval, the higher the score. The calculation formula is: T-Score = e^(-Δt / λ), where Δt is the time interval and λ is the decay coefficient, which is 7 days.

[0019] Furthermore, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak data operation period and high-frequency access data corresponding to the business scenario. A scheduled task is set to load the high-frequency access data into the high-speed memory before the peak data operation period arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The cold and hot data identification values ​​of the data are updated in real time using monitoring nodes. When the access frequency of cold data suddenly increases, the data upgrade process is triggered to adjust the cold data into warm data and load it into the medium-speed memory.

[0020] This ensures business continuity, reduces access latency, and prevents core business data from being replaced.

[0021] Furthermore, the decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. The roles of memory nodes include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, automatically incremented by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure the uniqueness of the time series; and PrevVer is the predecessor version number of the current version. A version chain of data modifications is constructed to achieve historical data traceability.

[0022] Furthermore, the operation request type is either a read operation or a write operation. For a read operation, if reading the latest version of data, the user terminal uses the latest version data from its local memory node. If reading historical version data, the user terminal requests the fragment location of the historical version data from the scheduling node based on the version vector, and obtains the historical version data based on the fragment location. For a write operation, before initiating a write request, the user terminal first obtains the version vector of the current data fragment, generates a new version number, completes the data modification locally, and asynchronously sends a write request + new version vector to the associated scheduling node. After receiving the request, the scheduling node verifies the legality of the modification through a majority voting mechanism, and simultaneously compares the version vectors to determine if there is a conflict. If there is no conflict, consistency confirmation is completed.

[0023] Furthermore, the cross-node data interaction algorithm for Remote Direct Memory Access (RDMA) is as follows: A topology-aware data fragmentation routing strategy is constructed. Monitoring nodes periodically collect cluster network topology data, including the rack to which the memory node belongs, switch connection relationships, and link bandwidth utilization, to construct a topology matrix. Based on this matrix, nodes within the same rack are prioritized as data interaction targets, followed by nodes in different racks within the same switch, and finally nodes across switches. When link bandwidth is overloaded or node load is too high, the algorithm automatically switches to a backup routing path. A zero-copy batch data transmission mechanism is implemented, where memory nodes pre-register their local memory regions and map the data fragments to be transmitted as physical memory pages accessible by RDMA, avoiding data copying between kernel and user modes. For large volumes of data in power grid infrastructure, a block-based batch transmission strategy is adopted, dividing the data into fixed-size blocks and sending them in batches via a reliable RDMA connection, reducing the number of handshakes. After receiving the data block, the receiving memory node verifies data integrity using a CRC32 checksum algorithm. If data loss or errors occur, only the abnormal data block is retransmitted, rather than all data, improving transmission efficiency.

[0024] Furthermore, the present invention also proposes a computer-readable storage medium storing computer program code, which, when executed by a computer, performs any of the methods described above.

[0025] The technical advantages of this invention are as follows: This invention provides a distributed memory data management method, apparatus, and storage medium for power grid infrastructure projects. The method includes: a storage step S101, where multi-source power grid infrastructure data collected by acquisition nodes is cleaned and stored in memory nodes directly connected to the acquisition nodes, and a hot / cold data hierarchical memory scheduling algorithm is used to classify and hierarchically store hot and cold data in all memory nodes; a request step S102, where a user terminal receives user operation requests, and a scheduling node locates the memory node corresponding to the operation request based on a decentralized consistency protocol according to the type of the operation request; a judgment step S103, where it is determined whether the operation corresponding to the operation request is a cross-memory node operation; if so, a cross-node data interaction algorithm based on remote direct memory access is triggered to prepare for cross-node data interaction; and a feedback step S104, where the corresponding operation is completed based on the type of the operation request, and the processing result is fed back to the user. This invention proposes a distributed memory data management method for power grid infrastructure scenarios, combining the characteristics of power grid infrastructure data services, and solves the consistency, performance, and reliability problems of distributed memory data management in power grid infrastructure business scenarios. Attached Figure Description

[0026] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart of a distributed memory data management method for a power grid infrastructure project according to an embodiment of the present invention.

[0028] Figure 2 This is a structural diagram of a distributed memory data management device for a power grid infrastructure project according to an embodiment of the present invention. Detailed Implementation

[0029] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Figure 1 This invention illustrates a distributed memory data management method for power grid infrastructure projects, the method comprising: In storage step S101, the multi-source data of power grid infrastructure collected by the acquisition node is cleaned and stored in the memory node directly connected to the acquisition node. The cold and hot data hierarchical memory scheduling algorithm is used to complete the classification of cold and hot data in all memory nodes and the hierarchical storage of memory. Request step S102: Use the user terminal to receive the user's operation request, and use the scheduling node to locate the memory node corresponding to the operation request based on the type of the operation request using a decentralized consensus protocol; In step S103, it is determined whether the operation corresponding to the operation request is a cross-memory node operation. If so, the cross-node data interaction algorithm based on remote direct memory access is triggered to prepare for cross-node data interaction. In feedback step S104, the corresponding operation is completed based on the type of the operation request, and the processing result is fed back to the user.

[0032] This invention addresses the four major data processing requirements of high-concurrency read / write, strong consistency guarantee, low-latency cross-node interaction, and efficient memory utilization for power grid infrastructure projects throughout their entire lifecycle. It proposes a distributed memory data management method for power grid infrastructure scenarios, combining the characteristics of power grid infrastructure data services. This method overcomes the bottleneck of insufficient adaptability of traditional algorithms from three dimensions: consistency coordination, memory scheduling, and cross-node interaction, thereby constructing a highly reliable data management capability that is adaptable to multiple disciplines and the entire process of power grid infrastructure.

[0033] To address the shortcomings of existing technologies, this invention first employs a hot and cold data hierarchical memory scheduling algorithm to classify and store hot and cold data across all memory nodes in a hierarchical manner. Then, it uses the same algorithm to complete the classification and hierarchical storage of hot and cold data across all memory nodes. Next, it determines whether the operation corresponding to the request is a cross-memory node operation. If so, it triggers a cross-node data interaction algorithm based on remote direct memory access to prepare for cross-node data interaction. In other words, this invention, through the synergistic cooperation of the hot and cold data hierarchical memory scheduling algorithm, the decentralized consistency protocol, and the cross-node data interaction algorithm based on remote direct memory access, fully adapts to the characteristics of power grid infrastructure business scenarios and solves the consistency, performance, and reliability problems of distributed memory data management in power grid infrastructure business scenarios. This is one of the key inventive concepts of this invention.

[0034] Power grid infrastructure data exhibits a significant hot-cold data stratification: operational data such as construction progress, safety monitoring, and equipment arrival are considered hot data (high-frequency access, high real-time requirements); while as-built documentation, audit reports, and historical acceptance data are considered cold data (low-frequency access, long-term retention requirements). Traditional memory scheduling algorithms (such as simple LRU) do not consider operational characteristics, easily leading to the replacement of hot data and wasted memory resources. Therefore, a hierarchical memory scheduling algorithm is designed to achieve refined utilization of memory resources.

[0035] In one embodiment, the present invention designs the hot and cold data hierarchical memory scheduling algorithm as follows: constructing hot and cold data identification values ​​based on three dimensions of business weight, access frequency, and access time; achieving data classification through quantitative scoring; dividing all memory nodes into three levels according to read and write speed: high-speed memory, medium-speed memory, and low-speed cache, corresponding to the storage of hot data, warm data, and cold data, thereby realizing differentiated storage and scheduling of data with different priorities; and constructing a business-aware memory preloading and replacement strategy based on the historical data access time sequence characteristics of power grid infrastructure business.

[0036] In one embodiment, the Hot-Score calculation formula for the hot / cold data identification value is: Hot-Score =α×F-Score +β×B-Score +γ×T-Score; Where α+β+γ=1, the weights are dynamically adjusted according to the business scenario, with a default α=0.4, β=0.3, and γ=0.3; a scoring threshold is set: Hot-Score ≥ 0.7 is considered hot data, and 0.3 ≤ Hot-Score 0.7 indicates warm data, Hot-Score 0.3 is considered cold data; F-Score is an access frequency score, which counts the number of times a data shard is accessed in the past 7 days and is divided into 5 levels according to the percentile. The higher the level, the higher the score. B-Score is a business weight score, which is assigned based on the priority of power grid infrastructure business. Progress management and safety management data are assigned a value of 0.9~1.0, quality management data is assigned a value of 0.7~0.9, cost management and material management data is assigned a value of 0.5~0.7, and historical archived data is assigned a value of 0.1~0.5. T-Score is the access time score, which is the interval between the last access to the data and the current time. The shorter the interval, the higher the score. The calculation formula is: T-Score = e^(-Δt / λ), where Δt is the time interval and λ is the decay coefficient, which is 7 days.

[0037] Hot data resides in high-speed memory: Hot data is directly loaded into DRAM / HBM, using a full memory storage mode to eliminate disk I / O latency and support millisecond-level queries; a memory waterline (e.g., 80%) is set, and when the remaining capacity of high-speed memory falls below the waterline, a warm data degradation process is triggered; Warm data is cached in medium-speed memory: Warm data is stored in PMem, using a cache storage mode, and access is directly read from medium-speed memory, with latency improved by more than 80% compared to disk reads; when high-speed memory is expanded, warm data is automatically upgraded to high-speed memory; Cold data is archived in a low-speed cache layer: Cold data is stored in SSD, using an on-demand loading mode, and is only loaded into medium-speed memory when a user initiates an access request. After the access is completed, a cold / hot score is used to determine whether to retain it in medium-speed memory, avoiding invalid data occupying memory resources.

[0038] This invention constructs a hot and cold data identification method based on three dimensions: business weight, access frequency, and access time. It achieves data classification through quantitative scoring, avoiding the limitations of single-indicator judgment. Furthermore, it proposes a Hot-Score calculation formula for the hot and cold data identification value. This formula considers the weight of business scenarios, thereby solving the defects of existing memory scheduling algorithms that do not combine business and are prone to replacing hot data and wasting memory resources. This enables refined utilization of memory resources, which is another important inventive concept of this invention.

[0039] In one embodiment, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak data operation period and high-frequency access data corresponding to the business scenario. A scheduled task is set to load the high-frequency access data into the high-speed memory before the peak data operation period arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The cold and hot data identification values ​​of the data are updated in real time using monitoring nodes. When the access frequency of cold data suddenly increases, the data upgrade process is triggered, and the cold data is adjusted to warm data and loaded into the medium-speed memory.

[0040] In this invention, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak period of data operation and high-frequency access data corresponding to the business scenario (such as the engineering acceptance stage or key construction node). A timed task is set to load the high-frequency access data into the high-speed memory before the peak period of data operation arrives, thereby ensuring business continuity, reducing access latency, and preventing core business data from being replaced. This is another important inventive concept of this invention.

[0041] In one embodiment, this invention addresses the shortcomings of traditional centralized consistency protocols (such as Raft) in multi-node clusters of power grid infrastructure, including single-point failure risk, high write latency under high concurrency, and overload of coordinating nodes. It proposes a centralized consistency protocol with the core objective of reducing the pressure on coordinating nodes and improving transaction processing efficiency in high-concurrency scenarios while ensuring strong data consistency. The decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. These memory node roles include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, incrementing by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure time series uniqueness; and PrevVer is the predecessor version number of the current version. A version chain for data modification is constructed to enable historical data tracing.

[0042] Data nodes are responsible for storing fragmented data, executing local computing tasks, and responding to read and write requests. Each data node corresponds to several power grid infrastructure data fragments (such as construction progress data of a certain section, equipment ledger data, etc.) and is the core carrier of data processing. Scheduling nodes: responsible for data shard location, consistency coordination, and fault detection. The number of coordination nodes dynamically scales according to the cluster size. They are bound to data shards through a consistent hashing algorithm to avoid load bottlenecks caused by fixed coordination nodes. Monitoring nodes: Independent of data and coordination tasks, they collect real-time data on cluster node status, data access frequency, transaction execution efficiency, and other metrics, providing decision-making basis for coordination node scheduling and memory allocation, and ensuring stable cluster operation.

[0043] This invention addresses the shortcomings of fixed node roles in business scenario data processing by proposing a dynamic allocation and balanced scheduling of node roles based on node performance indicators (CPU utilization, remaining memory capacity, network bandwidth) and business data weights. The invention also proposes a specific calculation method that demonstrates the important role of business data weights in node role allocation, thereby making node allocation more in line with the business scenarios of power grid infrastructure. This is another inventive concept of this invention.

[0044] The method for dynamically adjusting the role of memory nodes based on their performance metrics and business data weights is as follows: Calculate the Role-Score of a memory node: ; Among them, Pcpu represents CPU availability, Pmem represents memory availability, PnetW represents network bandwidth availability, B-Score is the service data weight score, and a Role-Score greater than or equal to the first threshold is used as a monitoring node, a Role-Score less than the second threshold is used as a data node, and the rest are used as scheduling nodes.

[0045] In one embodiment, the operation request type is either a read operation or a write operation. For a read operation, if reading the latest version of data, the user terminal uses the latest version data from its local memory node; if reading historical version data, the user terminal requests the fragment location of the historical version data from the scheduling node based on the version vector, and obtains the historical version data based on the fragment location. For a write operation, before initiating a write request, the user terminal first obtains the version vector of the current data fragment, generates a new version number, completes the data modification locally, and asynchronously sends a write request + new version vector to the associated scheduling node. After receiving the request, the scheduling node verifies the legality of the modification through a majority voting mechanism and compares the version vectors to determine if there is a conflict. If there is no conflict, consistency confirmation is completed.

[0046] This invention also addresses the need for 24 / 7 continuous operation in power grid infrastructure projects by designing an automatic fault recovery process based on heartbeat detection and replica migration to ensure no data loss and uninterrupted service. 1. Fault detection: The monitoring node and the coordination node periodically send heartbeat packets to the target node. If no response is received for three consecutive heartbeat cycles, the node is considered faulty. 2. Replica Location: Based on the data sharding replication strategy (default 3 replicas, stored on different rack nodes), locate the backup replica node of the failed node; 3. Task Migration: The coordinating node automatically migrates the data sharding coordination tasks and computation tasks of the failed node to the standby replica node, while updating the node identifier in the version vector; 4. Data Completion: The standby replica node completes the incremental data that was not synchronized during the failure period based on the local replica data and the cluster operation logs collected by the monitoring node, thus completing the data consistency recovery.

[0047] Power grid infrastructure data exhibits strong correlation characteristics (e.g., BIM model data, equipment ledgers, and construction progress data require cross-node correlation queries). Traditional cross-node data transmission under the TCP / IP protocol suffers from high CPU usage, large transmission latency, and low bandwidth utilization, severely impacting the efficiency of multi-node parallel computing. Therefore, this invention proposes a cross-node data interaction algorithm based on Remote Direct Memory Access (RDMA) to achieve zero-copy, low-latency data transmission, which is one of the key inventive concepts of this invention.

[0048] In one embodiment, the cross-node data interaction algorithm for Remote Direct Memory Access (RDMA) is as follows: A topology-aware data fragmentation routing strategy is constructed. Monitoring nodes periodically collect cluster network topology data, including the rack to which the memory node belongs, switch connection relationships, and link bandwidth utilization. A topology matrix is ​​constructed. Based on the topology matrix, nodes within the same rack are preferentially selected as data interaction targets, followed by nodes in different racks within the same switch, and finally nodes across switches. When link bandwidth is overloaded or node load is too high, the algorithm automatically switches to a backup routing path. A zero-copy batch data transmission mechanism is implemented. Memory nodes pre-register their local memory regions, mapping the data fragments to be transmitted to physical memory pages accessible by RDMA, avoiding data copying between kernel and user modes. For large-volume data in power grid infrastructure, a block-based batch transmission strategy is adopted, dividing the data into fixed-size blocks and sending them in batches via a reliable RDMA connection, reducing the number of handshakes. After receiving the data block, the receiving memory node verifies the data integrity using a CRC32 checksum algorithm. If data loss or errors occur, only the abnormal data block is retransmitted, not all data. This invention addresses the network topology of multi-rack, multi-node power grid infrastructure clusters by designing a topology-aware routing algorithm to optimize data transmission paths, reduce cross-rack transmission latency, and improve transmission efficiency. This is another important inventive concept of this invention.

[0049] Figure 2This invention discloses a distributed memory data management device for a power grid infrastructure project, the device comprising: Storage unit 201 cleans the multi-source power grid infrastructure data collected by the acquisition node and stores it in a memory node directly connected to the acquisition node. It also uses a hot and cold data hierarchical memory scheduling algorithm to classify and store hot and cold data in all memory nodes in a hierarchical memory manner. Request unit 202: Receives user operation requests using a user terminal, and locates the memory node corresponding to the operation request using a scheduling node based on a decentralized consensus protocol according to the type of the operation request; The judgment unit 203 determines whether the operation corresponding to the operation request is a cross-memory node operation. If so, it triggers a cross-node data interaction algorithm based on remote direct memory access to prepare for cross-node data interaction. Feedback unit 204 performs the corresponding operation based on the type of the operation request and feeds back the processing result to the user.

[0050] This invention addresses the four major data processing requirements of high-concurrency read / write, strong consistency guarantee, low-latency cross-node interaction, and efficient memory utilization for power grid infrastructure projects throughout their entire lifecycle. It proposes a distributed memory data management method for power grid infrastructure scenarios, combining the characteristics of power grid infrastructure data services. This method overcomes the bottleneck of insufficient adaptability of traditional algorithms from three dimensions: consistency coordination, memory scheduling, and cross-node interaction, thereby constructing a highly reliable data management capability that is adaptable to multiple disciplines and the entire process of power grid infrastructure.

[0051] To address the shortcomings of existing technologies, this invention first employs a hot and cold data hierarchical memory scheduling algorithm to classify and store hot and cold data across all memory nodes in a hierarchical manner. Then, it uses the same algorithm to complete the classification and hierarchical storage of hot and cold data across all memory nodes. Next, it determines whether the operation corresponding to the request is a cross-memory node operation. If so, it triggers a cross-node data interaction algorithm based on remote direct memory access to prepare for cross-node data interaction. In other words, this invention, through the synergistic cooperation of the hot and cold data hierarchical memory scheduling algorithm, the decentralized consistency protocol, and the cross-node data interaction algorithm based on remote direct memory access, fully adapts to the characteristics of power grid infrastructure business scenarios and solves the consistency, performance, and reliability problems of distributed memory data management in power grid infrastructure business scenarios. This is one of the key inventive concepts of this invention.

[0052] Power grid infrastructure data exhibits a significant hot-cold data stratification: operational data such as construction progress, safety monitoring, and equipment arrival are considered hot data (high-frequency access, high real-time requirements); while as-built documentation, audit reports, and historical acceptance data are considered cold data (low-frequency access, long-term retention requirements). Traditional memory scheduling algorithms (such as simple LRU) do not consider operational characteristics, easily leading to the replacement of hot data and wasted memory resources. Therefore, a hierarchical memory scheduling algorithm is designed to achieve refined utilization of memory resources.

[0053] In one embodiment, the present invention designs the hot and cold data hierarchical memory scheduling algorithm as follows: constructing hot and cold data identification values ​​based on three dimensions of business weight, access frequency, and access time; achieving data classification through quantitative scoring; dividing all memory nodes into three levels according to read and write speed: high-speed memory, medium-speed memory, and low-speed cache, corresponding to the storage of hot data, warm data, and cold data, thereby realizing differentiated storage and scheduling of data with different priorities; and constructing a business-aware memory preloading and replacement strategy based on the historical data access time sequence characteristics of power grid infrastructure business.

[0054] In one embodiment, the Hot-Score calculation formula for the hot / cold data identification value is: Hot-Score =α×F-Score +β×B-Score +γ×T-Score; Where α+β+γ=1, the weights are dynamically adjusted according to the business scenario, with a default α=0.4, β=0.3, and γ=0.3; a scoring threshold is set: Hot-Score ≥ 0.7 is considered hot data, and 0.3 ≤ Hot-Score 0.7 indicates warm data, Hot-Score 0.3 is considered cold data; F-Score is an access frequency score, which counts the number of times a data shard is accessed in the past 7 days and is divided into 5 levels according to the percentile. The higher the level, the higher the score. B-Score is a business weight score, which is assigned based on the priority of power grid infrastructure business. Progress management and safety management data are assigned a value of 0.9~1.0, quality management data is assigned a value of 0.7~0.9, cost management and material management data is assigned a value of 0.5~0.7, and historical archived data is assigned a value of 0.1~0.5. T-Score is the access time score, which is the interval between the last access to the data and the current time. The shorter the interval, the higher the score. The calculation formula is: T-Score = e^(-Δt / λ), where Δt is the time interval and λ is the decay coefficient, which is 7 days.

[0055] Hot data resides in high-speed memory: Hot data is directly loaded into DRAM / HBM, using a full memory storage mode to eliminate disk I / O latency and support millisecond-level queries; a memory waterline (e.g., 80%) is set, and when the remaining capacity of high-speed memory falls below the waterline, a warm data degradation process is triggered; Warm data is cached in medium-speed memory: Warm data is stored in PMem, using a cache storage mode, and access is directly read from medium-speed memory, with latency improved by more than 80% compared to disk reads; when high-speed memory is expanded, warm data is automatically upgraded to high-speed memory; Cold data is archived in a low-speed cache layer: Cold data is stored in SSD, using an on-demand loading mode, and is only loaded into medium-speed memory when a user initiates an access request. After the access is completed, a cold / hot score is used to determine whether to retain it in medium-speed memory, avoiding invalid data occupying memory resources.

[0056] This invention constructs a hot and cold data identification method based on three dimensions: business weight, access frequency, and access time. It achieves data classification through quantitative scoring, avoiding the limitations of single-indicator judgment. Furthermore, it proposes a Hot-Score calculation formula for the hot and cold data identification value. This formula considers the weight of business scenarios, thereby solving the defects of existing memory scheduling algorithms that do not combine business and are prone to replacing hot data and wasting memory resources. This enables refined utilization of memory resources, which is another important inventive concept of this invention.

[0057] In one embodiment, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak data operation period and high-frequency access data corresponding to the business scenario. A scheduled task is set to load the high-frequency access data into the high-speed memory before the peak data operation period arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The cold and hot data identification values ​​of the data are updated in real time using monitoring nodes. When the access frequency of cold data suddenly increases, the data upgrade process is triggered, and the cold data is adjusted to warm data and loaded into the medium-speed memory.

[0058] In this invention, the business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak period of data operation and high-frequency access data corresponding to the business scenario (such as the engineering acceptance stage or key construction node). A timed task is set to load the high-frequency access data into the high-speed memory before the peak period of data operation arrives, thereby ensuring business continuity, reducing access latency, and preventing core business data from being replaced. This is another important inventive concept of this invention.

[0059] In one embodiment, this invention addresses the shortcomings of traditional centralized consistency protocols (such as Raft) in multi-node clusters of power grid infrastructure, including single-point failure risk, high write latency under high concurrency, and overload of coordinating nodes. It proposes a centralized consistency protocol with the core objective of reducing the pressure on coordinating nodes and improving transaction processing efficiency in high-concurrency scenarios while ensuring strong data consistency. The decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. These memory node roles include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, incrementing by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure time series uniqueness; and PrevVer is the predecessor version number of the current version. A version chain for data modification is constructed to enable historical data tracing.

[0060] Data nodes are responsible for storing fragmented data, executing local computing tasks, and responding to read and write requests. Each data node corresponds to several power grid infrastructure data fragments (such as construction progress data of a certain section, equipment ledger data, etc.) and is the core carrier of data processing. Scheduling nodes: responsible for data shard location, consistency coordination, and fault detection. The number of coordination nodes dynamically scales according to the cluster size. They are bound to data shards through a consistent hashing algorithm to avoid load bottlenecks caused by fixed coordination nodes. Monitoring nodes: Independent of data and coordination tasks, they collect real-time data on cluster node status, data access frequency, transaction execution efficiency, and other metrics, providing decision-making basis for coordination node scheduling and memory allocation, and ensuring stable cluster operation.

[0061] This invention addresses the shortcomings of fixed node roles in business scenario data processing by proposing a dynamic allocation and balanced scheduling of node roles based on node performance indicators (CPU utilization, remaining memory capacity, network bandwidth) and business data weights. The invention also proposes a specific calculation method that demonstrates the important role of business data weights in node role allocation, thereby making node allocation more in line with the business scenarios of power grid infrastructure. This is another inventive concept of this invention.

[0062] The method for dynamically adjusting the role of memory nodes based on their performance metrics and business data weights is as follows: Calculate the Role-Score of a memory node: ; Among them, Pcpu represents CPU availability, Pmem represents memory availability, PnetW represents network bandwidth availability, B-Score is the service data weight score, and a Role-Score greater than or equal to the first threshold is used as a monitoring node, a Role-Score less than the second threshold is used as a data node, and the rest are used as scheduling nodes.

[0063] In one embodiment, the operation request type is either a read operation or a write operation. For a read operation, if reading the latest version of data, the user terminal uses the latest version data from its local memory node; if reading historical version data, the user terminal requests the fragment location of the historical version data from the scheduling node based on the version vector, and obtains the historical version data based on the fragment location. For a write operation, before initiating a write request, the user terminal first obtains the version vector of the current data fragment, generates a new version number, completes the data modification locally, and asynchronously sends a write request + new version vector to the associated scheduling node. After receiving the request, the scheduling node verifies the legality of the modification through a majority voting mechanism and compares the version vectors to determine if there is a conflict. If there is no conflict, consistency confirmation is completed.

[0064] This invention also addresses the need for 24 / 7 continuous operation in power grid infrastructure projects by designing an automatic fault recovery process based on heartbeat detection and replica migration to ensure no data loss and uninterrupted service. 1. Fault detection: The monitoring node and the coordination node periodically send heartbeat packets to the target node. If no response is received for three consecutive heartbeat cycles, the node is considered faulty. 2. Replica Location: Based on the data sharding replication strategy (default 3 replicas, stored on different rack nodes), locate the backup replica node of the failed node; 3. Task Migration: The coordinating node automatically migrates the data sharding coordination tasks and computation tasks of the failed node to the standby replica node, while updating the node identifier in the version vector; 4. Data Completion: The standby replica node completes the incremental data that was not synchronized during the failure period based on the local replica data and the cluster operation logs collected by the monitoring node, thus completing the data consistency recovery.

[0065] Power grid infrastructure data exhibits strong correlation characteristics (e.g., BIM model data, equipment ledgers, and construction progress data require cross-node correlation queries). Traditional cross-node data transmission under the TCP / IP protocol suffers from high CPU usage, large transmission latency, and low bandwidth utilization, severely impacting the efficiency of multi-node parallel computing. Therefore, this invention proposes a cross-node data interaction algorithm based on Remote Direct Memory Access (RDMA) to achieve zero-copy, low-latency data transmission, which is one of the key inventive concepts of this invention.

[0066] In one embodiment, the cross-node data interaction algorithm for Remote Direct Memory Access (RDMA) is as follows: A topology-aware data fragmentation routing strategy is constructed. Monitoring nodes periodically collect cluster network topology data, including the rack to which the memory node belongs, switch connection relationships, and link bandwidth utilization. A topology matrix is ​​constructed. Based on the topology matrix, nodes within the same rack are preferentially selected as data interaction targets, followed by nodes in different racks within the same switch, and finally nodes across switches. When link bandwidth is overloaded or node load is too high, the algorithm automatically switches to a backup routing path. A zero-copy batch data transmission mechanism is implemented. Memory nodes pre-register their local memory regions, mapping the data fragments to be transmitted to physical memory pages accessible by RDMA, avoiding data copying between kernel and user modes. For large-volume data in power grid infrastructure, a block-based batch transmission strategy is adopted, dividing the data into fixed-size blocks and sending them in batches via a reliable RDMA connection, reducing the number of handshakes. After receiving the data block, the receiving memory node verifies the data integrity using a CRC32 checksum algorithm. If data loss or errors occur, only the abnormal data block is retransmitted, not all data. This invention addresses the network topology of multi-rack, multi-node power grid infrastructure clusters by designing a topology-aware routing algorithm to optimize data transmission paths, reduce cross-rack transmission latency, and improve transmission efficiency. This is another important inventive concept of this invention.

[0067] One embodiment of the present invention provides a computer storage medium storing a computer program. When the computer program on the computer storage medium is executed by a processor, the above-described method is implemented. The computer storage medium may be a hard disk, DVD, CD, flash memory, or other storage device.

[0068] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0069] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the apparatus described in various embodiments or some parts of the embodiments of this application.

[0070] Finally, it should be noted that the above embodiments are for illustration only and not for limiting the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention without departing from the spirit and scope of the present invention. Any modifications or partial substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A distributed memory data management method for power grid infrastructure projects, characterized in that, The method includes: The storage step involves cleaning the multi-source power grid infrastructure data collected by the acquisition nodes and storing it in memory nodes directly connected to the acquisition nodes. A cold and hot data hierarchical memory scheduling algorithm is used to classify and store cold and hot data in all memory nodes in a hierarchical memory manner. Request steps: Use a user terminal to receive the user's operation request, and use a scheduling node to locate the memory node corresponding to the operation request based on the type of the operation request and a decentralized consensus protocol; The judgment step is to determine whether the operation corresponding to the operation request is a cross-memory node operation. If so, the cross-node data interaction algorithm based on remote direct memory access is triggered to prepare for cross-node data interaction. The feedback step involves performing the corresponding operation based on the type of the operation request and then feeding back the processing result to the user.

2. The method according to claim 1, characterized in that, The hot and cold data hierarchical memory scheduling algorithm is as follows: It constructs a hot and cold data identification value based on three dimensions: business weight, access frequency, and access time. It achieves data classification through quantitative scoring and divides all memory nodes into three levels: high-speed memory, medium-speed memory, and low-speed cache according to read and write speed. These levels store hot data, warm data, and cold data respectively, realizing differentiated storage and scheduling of data with different priorities. Based on the historical data access time sequence characteristics of power grid infrastructure business, it constructs a business-aware memory preloading and replacement strategy.

3. The method according to claim 2, characterized in that, The business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak period of data operation and high-frequency access data corresponding to the business scenario. A timed task is set to load the high-frequency access data into the high-speed memory before the peak period of data operation arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The monitoring node updates the hot and cold data identification values ​​in real time. When the access frequency of cold data suddenly increases, the data upgrade process is triggered to adjust the cold data to warm data and load it into the medium-speed memory.

4. The method according to claim 3, characterized in that, The decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. These memory node roles include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, incrementing by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure time series uniqueness; and PrevVer is the predecessor version number of the current version. A version chain for data modification is constructed to enable historical data tracing.

5. The method according to claim 4, characterized in that, The operation request type is either a read operation or a write operation. For a read operation, if reading the latest version of data, the user terminal uses the latest version data from its local memory node. If reading historical version data, the user terminal requests the fragment location of the historical version data from the scheduling node based on the version vector, and obtains the historical version data based on the fragment location. For a write operation, before initiating a write request, the user terminal first obtains the version vector of the current data fragment, generates a new version number, completes the data modification locally, and asynchronously sends a write request + new version vector to the associated scheduling node. After receiving the request, the scheduling node verifies the legality of the modification through a majority voting mechanism and compares the version vectors to determine if there is a conflict. If there is no conflict, consistency confirmation is completed.

6. A distributed memory data management device for power grid infrastructure projects, characterized in that, The device includes: The storage unit cleans the multi-source power grid infrastructure data collected by the acquisition node and stores it in the memory node directly connected to the acquisition node. It also uses a hot and cold data hierarchical memory scheduling algorithm to classify and store hot and cold data in all memory nodes. Request Unit: Receives user operation requests using a user terminal, and locates the memory node corresponding to the operation request using a scheduling node based on a decentralized consensus protocol, according to the type of the operation request. The judgment unit determines whether the operation corresponding to the operation request is a cross-memory node operation. If so, it triggers a cross-node data interaction algorithm based on remote direct memory access to prepare for cross-node data interaction. The feedback unit performs the corresponding operation based on the type of the operation request and feeds back the processing result to the user.

7. The apparatus according to claim 6, characterized in that, The hot and cold data hierarchical memory scheduling algorithm is as follows: It constructs a hot and cold data identification value based on three dimensions: business weight, access frequency, and access time. It achieves data classification through quantitative scoring and divides all memory nodes into three levels: high-speed memory, medium-speed memory, and low-speed cache according to read and write speed. These levels store hot data, warm data, and cold data respectively, realizing differentiated storage and scheduling of data with different priorities. Based on the historical data access time sequence characteristics of power grid infrastructure business, it constructs a business-aware memory preloading and replacement strategy.

8. The apparatus according to claim 7, characterized in that, The business scenario corresponding to the operation request is input into the trained LSTM network to predict the peak period of data operation and high-frequency access data corresponding to the business scenario. A timed task is set to load the high-frequency access data into the high-speed memory before the peak period of data operation arrives. When the high-speed memory needs to be expanded, warm data with low business weight and low access frequency is replaced first, while hot data with high business weight is retained. The monitoring node updates the hot and cold data identification values ​​in real time. When the access frequency of cold data suddenly increases, the data upgrade process is triggered to adjust the cold data to warm data and load it into the medium-speed memory.

9. The apparatus according to claim 8, characterized in that, The decentralized consistency protocol includes: a dynamic adjustment mechanism for memory node roles, which dynamically adjusts the roles of memory nodes based on their performance metrics and business data weights. These memory node roles include: data nodes, scheduling nodes, and monitoring nodes. A version vector conflict detection strategy is constructed, assigning a unique version vector to each data shard to achieve lock-free concurrency control. The version vector is defined as: Version = {NodeID, VerNum, TimeStamp, PrevVer}, where: NodeID is the identifier of the node initiating the data modification; VerNum is the version sequence number of the data shard, initially 0, incrementing by 1 after each valid modification; TimeStamp is the global timestamp of the modification operation, synchronized with the system clock to ensure time series uniqueness; and PrevVer is the predecessor version number of the current version. A version chain for data modification is constructed to enable historical data tracing.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Images