Distributed real-time database high availability method, system and device of new energy centralized control station system and storage medium

Abstract

The invention discloses a distributed real-time database high availability method, system and device of a new energy centralized control station system and a storage medium. Based on a distributed architecture, a real-time data writing strategy, a real-time data reading strategy of the distributed architecture, a high-availability strategy after host exit of the distributed architecture and a high-availability strategy after host recovery of the distributed architecture are constructed, and respective storage nodes and hot standby nodes are created for a plurality of hosts to form a complete real-time library. According to the method, the acquisition, storage and reading of the large-capacity real-time database of the new energy centralized control station system are realized by utilizing load balancing, consistent hash, node heartbeat monitoring and failover, the high availability of real-time data is realized, and the reliability of the whole monitoring system is improved.

Description

Technical Field

[0001] This invention relates to the field of power system automation technology, and in particular to a method, system, device and storage medium for a distributed real-time database of a new energy central control station system. Background Technology

[0002] With the growth of cumulative installed photovoltaic (PV) capacity and energy storage system capacity, new energy power plants need to collect data at the second level for each PV panel (including parameters such as PV current, voltage, temperature, and irradiance) and energy storage cell (including parameters such as voltage, current, temperature, and SOC). In a typical scenario, a single 20MW PV power plant generates 1.2 million real-time data points per second, and an energy storage system generates 3.6 million data points per second, more than 100 times higher than traditional substations. Simultaneously, the data exhibits strong temporal sequence (sampling interval ≤100ms), high dimensionality (more than 200 parameters per device), and time-varying nonlinearity (power fluctuations of ±30% due to sudden changes in light intensity), posing stringent requirements for data storage and processing. Existing real-time power monitoring systems suffer from the following problems: 1. Standalone architecture: The system's real-time library is stored on a single machine, which can store hundreds of thousands of data entries, but cannot meet the real-time writing and reading requirements of millions of data entries.

[0003] 2. High load rate: All data is stored on a single machine, which needs to meet the access and writing requirements of various services and applications. This places high demands on the performance of the single machine and creates a single machine performance bottleneck.

[0004] 3. Insufficient scalability: To increase high availability, a second host is needed as a backup host to maintain a standby state for the original host's real-time database. However, this fails to address the bottleneck of writing millions of data points and single-machine performance when the primary host fails and switches to the backup host. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a method for high availability of a large-capacity distributed real-time database used in a new energy central control station system; another purpose of this invention is to provide a distributed real-time database high availability system, equipment, and storage medium for a new energy central control station system.

[0006] Technical solution: The distributed real-time database high availability method for the new energy centralized control station system of the present invention includes: (1) The distributed real-time database is equipped with at least three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host. (2) Real-time data writing strategy: Based on consistent hashing and dynamic weighted load balancing, data is distributed and written to the storage nodes of all hosts, and the data of the storage nodes is synchronously written to the hot standby nodes corresponding to the storage nodes. (3) Real-time data reading strategy: Data is located through consistent hashing, and data is read from hot standby nodes first and the results are aggregated; (4) Host exit high availability strategy: When the heartbeat monitoring detects a fault, the hot standby node corresponding to the storage node of the faulty host is upgraded to a new storage node, and the storage node of the faulty host is downgraded to a new hot standby node. (5) Host recovery high availability strategy: When the faulty host recovers and reconnects to the network, the three hosts are restored to parallel operation through data synchronization and node role conversion.

[0007] Furthermore, in the real-time data writing strategy, based on the written data ID, data alias, data value, data time, and quality, the data alias is used to generate a unique hash value for the entire site's data, which serves as the data index. The load balancer module reads the server's CPU, memory, and disk I / O parameters, calculates the host load weight, and prevents any host node from becoming overloaded. The load balancer allocates data and writes it to the corresponding host's storage node, and the data synchronization program writes the data from the storage node to the hot standby node.

[0008] Furthermore, host load weight for ; in, For the first k The weighting coefficient of each indicator ; For nodes The load value of the Kth indicator; The maximum threshold for the k-th indicator; n This represents the total number of indicators.

[0009] Furthermore, in the real-time data reading strategy, real-time data consists of data ID, data alias, data value, data time, and quality. The data alias is unique across all data on the site. The system uses the data alias to generate a consistent hash value, which serves as the basis for querying and retrieving data. When both the storage node and its corresponding hot standby node exist, the load balancer module prioritizes querying the hot standby node data; if the hot standby node does not exist, it queries the storage node data. Data is retrieved based on the hash value generated by the alias, then aggregated to remove data with the same hash value.

[0010] Furthermore, in the host exit high availability strategy, when it is detected that the communication status between host A and other hosts is abnormal, and other hosts also detect that the communication status with host A is abnormal, then it is determined that the communication between host A and host A is abnormal. Each host runs a failover procedure. When host A is found to be in communication failure, the storage node of host A is automatically converted into a hot standby node. The failover procedures on other hosts convert the hot standby node corresponding to host A into a storage node. All storage nodes in the system run simultaneously.

[0011] Furthermore, in the host recovery high availability strategy, when host A, which is in a communication abnormal state, reconnects to the network, and when it is detected that host A's communication with other hosts is normal, and other hosts also detect that their communication with host A is normal, then it is determined that host A's communication is normal. The failover procedure synchronizes the hot standby node data of host A from the corresponding storage nodes of other hosts, and upgrades the hot standby node role of host A to a storage node, while the corresponding storage nodes of other hosts are downgraded to hot standby nodes.

[0012] The distributed real-time database high-availability system of the new energy centralized control station system of the present invention includes: The real-time database storage pool building module is used for distributed real-time databases equipped with at least three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host. The real-time data writing module is used to distribute data to the storage nodes of all hosts based on consistent hashing and dynamic weighted load balancing, and to synchronously write the data of the storage nodes to the corresponding hot standby nodes. The real-time data reading module is used to locate data through consistent hashing, prioritize reading data from hot standby nodes, and aggregate the results. The monitoring module is used to detect node status, realize heartbeat monitoring between hosts, periodically send heartbeat packets to all nodes, realize TCP probe, and record node liveness status; The control module is used to upgrade the hot standby node corresponding to the storage node of the faulty host to a new storage node and downgrade the storage node of the faulty host to a new hot standby node when the heartbeat monitoring detects a fault. When the faulty host recovers and re-enters the network, it restores the parallel operation of the three hosts through data synchronization and node role conversion.

[0013] Furthermore, in the real-time data writing module, based on the written data ID, data alias, data value, data time, and quality, a unique hash value for the entire site's data is generated using the data alias, which serves as the data index. In the real-time data reading module, real-time data consists of data ID, data alias, data value, data time, and quality. The data alias is unique across all data on the site. The system uses the data alias to generate a consistent hash value, which serves as the basis for querying and retrieving data.

[0014] The computer device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0015] The computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] Beneficial effects: Compared with the prior art, the significant advantages of this invention are: 1. Based on the large-capacity data access of new energy power stations, this invention splits the real-time database into at least three storage nodes, while providing read and write services simultaneously. This maximizes the utilization of the resources of three hosts. After testing, it can meet the reading and writing of five million real-time data. 2. By introducing a data hot standby node, this invention ensures that after one host fails, the three storage nodes continue to operate normally without affecting the real-time data read and write services through a failover mechanism, thus achieving high availability of real-time data. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the real-time data writing strategy of the distributed architecture of the present invention; Figure 2 This is a schematic diagram of the real-time data reading strategy of the distributed architecture of the present invention; Figure 3 This is a schematic diagram of the high availability strategy after a host exits in the distributed architecture of this invention; Figure 4 This is a schematic diagram of the high availability strategy after host recovery in the distributed architecture of this invention. Detailed Implementation

[0018] The distributed real-time database high availability method for the new energy centralized control station system of the present invention includes: (1) The distributed real-time database is equipped with three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host.

[0019] Leveraging the distributed nature of multi-host architecture, a real-time library storage pool is created on three hosts, defined as storage nodes node1, node2, and node3. Millions of real-time data are stored in a distributed manner, stored on three storage nodes: node1, node2, and node3. The real-time library storage pool provides unified client data read and write services.

[0020] Hot standby nodes (such as node1-1) and storage nodes (such as node1) synchronize data in real time, supporting only redundant nodes for read requests; To prevent data loss due to the shutdown of any host, ensure that the hot standby node and its own storage node are not on the same host.

[0021] (2) Real-time data writing strategy: Based on consistent hashing and dynamic weighted load balancing, data is distributed and written to the storage nodes of all hosts, and the data of the storage nodes is synchronously written to the hot standby nodes corresponding to the storage nodes.

[0022] Based on data consistency hashing, load balancing analysis and data synchronization were performed on the three hosts, as detailed below: Based on the written data ID, data alias, data value, data time, and quality, a site-wide unique hash value is generated using the data alias, which serves as the data index. The load balancer module reads server CPU, memory, and disk I / O parameters, calculates host load weights, and prevents any host node from becoming overloaded. Load weight calculation method: ; in, For the first k The weighting coefficient of each indicator For example: CPU=0.4, Memory=0.3, Disk I / O=0.3; For nodes The Kth metric is the load value, which includes CPU utilization, memory usage, and disk I / O. This represents the maximum threshold for the k-th metric, such as CPU=80% or disk I / O=20ms. n This represents the total number of indicators.

[0023] The load balancer distributes data and writes it to the corresponding host's storage node. The data synchronization program writes the data from the storage node to the hot standby node, such as node1. The data synchronization program also writes the data from the storage node to the hot standby node, node1-1. The storage node and the hot standby node store the same data, and this synchronization is done automatically and in real-time. A storage node and its corresponding hot standby node with the same data cannot reside on the same host to ensure high system availability.

[0024] (3) Real-time data reading strategy: Data is located through consistent hashing, and data is read from hot standby nodes first and the results are aggregated.

[0025] Data consistency hashing, load balancing analysis of three hosts, and data integration are detailed below: Real-time data consists of data ID, data alias, data value, data time, and quality. Data aliases are unique across the entire site. The system uses data aliases to generate consistent hash values, which serve as the basis for querying and retrieving data.

[0026] When both the storage node and its corresponding hot standby node exist, the load balancer module prioritizes querying the hot standby node's data. If the hot standby node does not exist, it queries the storage node for data. For example, if both the storage node and its corresponding hot standby node exist, the load balancer module queries the hot standby node data for node1-1, node2-1, and node3-1 based on their aliases. If the hot standby node does not exist, it reads the data from the storage node.

[0027] Data is retrieved based on the hash value generated by the alias, then aggregated to remove data with the same hash value.

[0028] (4) Host exit high availability strategy: When the heartbeat monitoring detects a fault, the hot standby node corresponding to the storage node of the faulty host is upgraded to a new storage node, and the storage node of the faulty host is downgraded to a new hot standby node.

[0029] The host is equipped with a monitoring module to detect node status and implement heartbeat monitoring between hosts. It periodically sends heartbeat packets to all nodes to perform TCP probes and record node liveness status.

[0030] Each host runs a monitoring module to monitor the status between itself and other hosts. When the detection module of host 1 detects that the heartbeat messages with host 2 and host 3 are abnormal, host 2 detects that the heartbeat message of host 1 is abnormal, and the heartbeat message of host 3 is normal, or host 3 detects that the heartbeat message of host 1 is abnormal, but the heartbeat message of host 2 is normal, then host 1 is determined to be abnormal.

[0031] Host 1 includes a storage node (node1) and a hot standby node (node2-1). After host 1 exits, the failover procedure automatically downgrades the storage node (node1) of host 1 to a hot standby node (node1-1), while the hot standby node (node2-1) remains unchanged. The hot standby node (node1-1) of host 3 is upgraded to the storage node (node1) of host 3, ensuring that the system still has storage nodes (node1, node2, and node3) and that real-time data reading and writing are not affected.

[0032] When any host exits the system, the hot standby node is upgraded to a storage node, ensuring that node1, node2, and node3 are always online and running, and that the system's write and read services are not affected.

[0033] (5) Host recovery high availability strategy: When the faulty host recovers and reconnects to the network, the three hosts are restored to parallel operation through data synchronization and node role conversion.

[0034] When host 1 reconnects to the network, its monitoring module detects that the heartbeats with hosts 2 and 3 have returned to normal. Host 2's monitoring module also detects that the heartbeats with hosts 1 and 3 have returned to normal. Host 3's monitoring module also detects that the heartbeats with hosts 1 and 2 have returned to normal, thus determining that host 1 itself is functioning normally.

[0035] At this point, the failover procedure synchronizes the data of the node1-1 hot standby node of host 1 from the node1 storage node of host 3, and the data of the node2-1 hot standby node from the node2 of host 2. It then upgrades the node1-1 hot standby node of host 1 to the node1 storage node, and downgrades the node1 storage node of host 3 to the node1-1 hot standby node.

[0036] Ultimately, it is ensured that the system's storage nodes node1, node2, and node3, as well as the hot standby nodes node1-1, node2-1, and node3-1, are operating normally, and that the storage nodes and their corresponding hot standby nodes are not on the same host node.

[0037] Storage nodes support data reading and writing, hot standby nodes support reading, and storage nodes and hot standby nodes are synchronized in real time to improve the read and write characteristics of the system and achieve high availability of the real-time database.

[0038] The distributed real-time database high-availability system of the new energy centralized control station system of the present invention includes: The real-time database storage pool building module is used for distributed real-time databases equipped with at least three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host. The real-time data writing module is used to distribute data to the storage nodes of all hosts based on consistent hashing and dynamic weighted load balancing, and to synchronously write the data of the storage nodes to the corresponding hot standby nodes. The real-time data reading module is used to locate data through consistent hashing, prioritize reading data from hot standby nodes, and aggregate the results. The monitoring module is used to detect node status, realize heartbeat monitoring between hosts, periodically send heartbeat packets to all nodes, realize TCP probe, and record node liveness status; The control module is used to upgrade the hot standby node corresponding to the storage node of the faulty host to a new storage node and downgrade the storage node of the faulty host to a new hot standby node when the heartbeat monitoring detects a fault. When the faulty host recovers and re-enters the network, it restores the parallel operation of the three hosts through data synchronization and node role conversion.

[0039] Furthermore, in the real-time data writing module, based on the written data ID, data alias, data value, data time, and quality, a unique hash value for the entire site's data is generated using the data alias, which serves as the data index. In the real-time data reading module, real-time data consists of data ID, data alias, data value, data time, and quality. The data alias is unique across all data on the site. The system uses the data alias to generate a consistent hash value, which serves as the basis for querying and retrieving data.

[0040] The computer device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0041] The computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

Claims

1. A method for high availability of distributed real-time database in a new energy centralized control station system, characterized in that, include: (1) The distributed real-time database is equipped with at least three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host. (2) Real-time data writing strategy: Based on consistent hashing and dynamic weighted load balancing, data is distributed and written to the storage nodes of all hosts, and the data of the storage nodes is synchronously written to the hot standby nodes corresponding to the storage nodes. (3) Real-time data reading strategy: Data is located through consistent hashing, and data is read from hot standby nodes first and the results are aggregated; (4) Host exit high availability strategy: When the heartbeat monitoring detects a fault, the hot standby node corresponding to the storage node of the faulty host is upgraded to a new storage node, and the storage node of the faulty host is downgraded to a new hot standby node. (5) Host recovery high availability strategy: When the faulty host recovers and reconnects to the network, the three hosts are restored to parallel operation through data synchronization and node role conversion.

2. The distributed real-time database high availability method for the new energy centralized control station system according to claim 1, characterized in that, In the real-time data writing strategy, based on the written data ID, data alias, data value, data time, and quality, the data alias is used to generate a unique hash value for the entire site's data, which serves as the data index. The load balancer module reads the server's CPU, memory, and disk I / O parameters, calculates the host load weight, and prevents any host node from becoming overloaded. The load balancer allocates data and writes it to the corresponding host's storage node, and the data synchronization program writes the data from the storage node to the hot standby node.

3. The distributed real-time database high availability method for the new energy centralized control station system according to claim 2, characterized in that, Host load weight for ； in, For the first k The weighting coefficient of each indicator ; For nodes The load value of the Kth indicator; The maximum threshold for the k-th indicator; n This represents the total number of indicators.

4. The distributed real-time database high availability method for the new energy centralized control station system according to claim 1, characterized in that, In the real-time data reading strategy, real-time data consists of data ID, data alias, data value, data time, and quality. The data alias is unique across the entire site. The system uses the data alias to generate a consistent hash value, which serves as the basis for querying and retrieving data. When both the storage node and the corresponding hot standby node exist, the load balancer module will prioritize querying the data of the hot standby node. If the hot standby node does not exist, query the storage node data; Data is retrieved based on the hash value generated by the alias, then aggregated to remove data with the same hash value.

5. The distributed real-time database high availability method for the new energy centralized control station system according to claim 1, characterized in that, In the host exit high availability strategy, when it is detected that the communication status between host A and other hosts is abnormal, and other hosts also detect that the communication status with host A is abnormal, then it is determined that the communication between host A and host A is abnormal. Each host runs a failover procedure. When host A is found to be in communication failure, the storage node of host A is automatically converted into a hot standby node. The failover procedures on other hosts convert the hot standby node corresponding to host A into a storage node. All storage nodes in the system run simultaneously.

6. The distributed real-time database high availability method for the new energy centralized control station system according to claim 1, characterized in that, In the host recovery high availability strategy, when host A, which is in a communication abnormal state, reconnects to the network, and when it is detected that host A's communication status with other hosts is normal, and other hosts also detect that their communication status with host A is normal, then it is determined that host A's communication is normal. The failover procedure synchronizes the hot standby node data of host A from the corresponding storage nodes of other hosts, and upgrades the hot standby node role of host A to a storage node, while the corresponding storage nodes of other hosts are downgraded to hot standby nodes.

7. A distributed real-time database high-availability system for a new energy centralized control station system, characterized in that, include: The real-time database storage pool building module is used for distributed real-time databases equipped with at least three hosts. Each host runs a storage node and a hot standby node to form a real-time database storage pool. The storage node and its corresponding hot standby node are not on the same host. The real-time data writing module is used to distribute data to the storage nodes of all hosts based on consistent hashing and dynamic weighted load balancing, and to synchronously write the data of the storage nodes to the corresponding hot standby nodes. The real-time data reading module is used to locate data through consistent hashing, prioritize reading data from hot standby nodes, and aggregate the results. The monitoring module is used to detect node status, realize heartbeat monitoring between hosts, periodically send heartbeat packets to all nodes, realize TCP probe, and record node liveness status; The control module is used to upgrade the hot standby node corresponding to the storage node of the faulty host to a new storage node and downgrade the storage node of the faulty host to a new hot standby node when the heartbeat monitoring detects a fault. When the faulty host recovers and re-enters the network, it restores the parallel operation of the three hosts through data synchronization and node role conversion.

8. The distributed real-time database high-availability system of the new energy centralized control station system according to claim 7, characterized in that, In the real-time data writing module, based on the written data ID, data alias, data value, data time, and quality, a unique hash value for the entire site's data is generated using the data alias, which serves as the data index. In the real-time data reading module, real-time data consists of data ID, data alias, data value, data time, and quality. The data alias is unique across all data on the site. The system uses the data alias to generate a consistent hash value, which serves as the basis for querying and retrieving data.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of any one of the methods described in claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images