Data high-speed access system and method in real-time monitoring of power grid

Abstract

The invention discloses a high-speed data access system and method in real-time monitoring of a power grid. The high-speed data access system comprises an scada host machine, a first-level cache service architecture and a second-level cache service architecture, data collected by the power micro-service is stored in the first-level cache service architecture, then the data is updated to the second-level cache service architecture, and finally the data is stored in the scada host machine; when the data processing service performs real-time library access, the first-level cache service is called in a dynamic library mode. According to the method, by introducing a first-level cache mechanism and a second-level cache mechanism, direct access to a real-time library is reduced, delay is reduced, the data access speed is increased, and the data access performance is optimized.

Description

Technical Field

[0001] The present invention relates to the field of cloud-edge integration and high-speed data access, and in particular to a system and method for high-speed data access in real-time monitoring of power grids. Background Art

[0002] The existing power grid cloud-edge fusion system is distributed and can be flexibly expanded, and adding shards can be quickly completed by starting microservices. The scada "bread" is cut horizontally to decouple the scada data and the services under the scada application, and to achieve the separation of business logic and data. The existing power grid cloud-edge fusion system needs to transform the scada application, which has a difficulty, that is, how to realize the adjustment and access of data by various scada microservices, that is, high-speed reading and writing of data.

[0003] like Figure 1 As shown, the existing implementation scheme mounts the real-time library file to the container when the container is started, reads the data through the real-time library local interface when reading, and writes the data in batches through the real-time library network interface when writing. The real-time library file of this scheme is mounted into the container, so that the data of the real-time library file is consistent between the host and the container. However, since the real-time library is implemented using the mmap file mapping memory mechanism. The host and the container themselves do not use namespaces, and the two are mapped to the shared memory in the system through mmap and are independent and isolated from each other. When updating the data of the real-time library, the shared memory is directly modified. If the shared memory data does not explicitly call the msync interface, it will be forced to synchronize to the file, otherwise the file will be updated regularly according to the dirty page and time mechanism of the operating system.

[0004] Since scada applications and containerization are maintained separately, that is, scada microservices need to be deployed on scada hosts, but one scada host cannot guarantee that there is no need to deploy a full amount of scada microservices. The data calculated by sca_cal and sca_fml_pubcal is written to sca01 through tablenet, but if the data is read through the local interface, the data of sca02 is read. At this time, the real-time library is required to ensure that the data of sca01 and sca02 are synchronized, but whether the synchronization period of the real-time library meets the business needs. Currently, sca_analog implements a batch write interface. According to the engineering experience of D5000 and the new generation, rtdb_server has efficiency problems when the load is relatively high. The specific reason is that the number of connections is too large due to too much concurrency. If the container is hot migrated, if it is not distributed storage, the mounting library may be abnormal. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a system and method for high-speed data access in real-time monitoring of power grids based on cloud-edge fusion, to solve the problems in the prior art, reduce direct access to real-time libraries, reduce latency, increase data access speed, and optimize data access performance.

[0006] Technical solution: The high-speed data access system for real-time monitoring of power grids described in the present invention includes a SCADA host, a first-level cache service architecture and a second-level cache service architecture; the data collected by the power microservice is stored in the first-level cache service architecture, and then the data is updated to the second-level cache service architecture, and finally stored in the SCADA host; when the data processing service performs real-time library access, the first-level cache service is called through a dynamic library.

[0007] Furthermore, the first-level cache service architecture includes a data storage module and a model synchronization module. The data storage module is used to store the real-time library model and data of the container. When the service is started, the model synchronization module sends a model synchronization downlink request to call the second-level cache service. The data storage module builds the cache of the data storage module through the model synchronization module.

[0008] The secondary cache service architecture includes a model library change monitoring service module, which is used to process model synchronization downlink requests of the primary cache service and monitor the download of real-time library table information.

[0009] Furthermore, the first-level cache service architecture includes an interface module, which includes a data reading interface and a data update interface; the data reading interface and the data update interface give priority to accessing the data storage module. If the data storage module does not have the model, that is, the page is missing, the model is cached to the data storage module by calling the model synchronization module.

[0010] Furthermore, the interface module also includes a data addition and deletion interface, which directly calls the secondary cache service to update the real-time library and sends the update notification to the internal model update notification channel.

[0011] Furthermore, the cache size of the data storage module supports dynamic scaling; the data storage module periodically checks whether the cache data has been updated. When the data has been updated, the cache data is updated and the updated data is pushed to the data update channel; when receiving data from the data update channel, the cache is updated after timestamp and node verification.

[0012] Furthermore, the secondary cache service architecture includes a data update processing service module, which is used to receive and process scada updated data in the container; receive data update channel data, and directly store non-measurement and status value data; use the time stamp storage module to perform time stamp verification on the received data for measurement and status value data, and store the data after the verification passes; and perform time stamp data storage on the update channel data; perform heterogeneous verification on the received measurement data, and if the verification fails, the data needs to be forwarded to the data snapshot channel; and regularly delete expired measurement data and measurement values ​​in the time stamp data storage.

[0013] Furthermore, the secondary cache service architecture includes a data snapshot processing service module, which is used for snapshot storage of heterogeneous data; receiving data snapshot channel data, storing the snapshot into the warehouse; and circularly deleting expired snapshot data.

[0014] In the method for high-speed data access in real-time monitoring of a power grid described in the present invention, the data collected by the power microservice is stored in a first-level cache service architecture, and then the data is updated to a second-level cache service architecture, and finally stored in a scada host machine; when the data processing service performs real-time library access, the first-level cache service is called by means of a dynamic library.

[0015] The electronic device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for high-speed data access in real-time monitoring of a power grid is implemented.

[0016] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the method for high-speed data access in real-time monitoring of a power grid is implemented.

[0017] Beneficial effects: Compared with the prior art, the advantages of the present invention are: the present invention reduces direct access to the real-time library, reduces latency, improves data access speed, and optimizes data access performance by introducing a first-level cache and a second-level cache mechanism. The cache mechanism supports dynamic data management, and can dynamically adjust the cache size and content according to the access frequency and timeliness of the data, optimizing the use of memory and storage resources. The first-level cache is directly integrated into the data processing service, and can capture and respond to frequent data access requests. This reduces the number of direct accesses to the underlying real-time library, thereby reducing the load on the database and avoiding potential performance bottlenecks. The second-level cache serves as a backup layer for data and mainly handles data write operations. This read-write separation design allows the system to efficiently handle data writes without affecting data read performance, further improving overall performance. The present invention adopts a cloud-edge fusion architecture, which significantly improves the flexibility and scalability of the real-time monitoring system for power grids. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the architecture diagram of the power grid cloud-edge fusion system based on the existing technology.

[0019] Figure 2 Schematic diagram of the principle architecture of a high-speed data access system according to an embodiment of the present invention.

[0020] Figure 3 This is a diagram of the high-speed data access system architecture of an embodiment of the present invention.

[0021] Figure 4 A schematic diagram of a first-level cache service architecture according to an embodiment of the present invention.

[0022] Figure 5 A schematic diagram of a secondary cache service architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, the high-speed data access system for real-time monitoring of the power grid includes a SCADA host, a first-level cache service architecture, and a second-level cache service architecture.

[0025] like Figure 2 The figure shows a schematic diagram of the principle architecture of the high-speed data access system of the present invention. The data processing service in the container accesses the real-time library data through the first-level cache service in the container. The data model in the first-level cache service is obtained by notifying the second-level cache service on the scada host to read the real-time library. When data needs to be updated during data processing, it is implemented through the first-level cache service, and the first-level cache service updates the updated data to its own cache. If it is a service host, a data update message needs to be sent to the data update channel. The message carries the data update timestamp and the update service name. After the first-level cache service in other containers receives the data update message, it compares it with the data update timestamp to determine whether the local cache data needs to be updated; after the second-level cache service receives the data update message, it compares it with the update timestamp of the local cache to determine whether the local real-time library and snapshot storage need to be updated.

[0026] The focus of the present invention is on fast data access. Caching is a technology that improves data access speed by storing frequently accessed data in faster storage media to reduce data access time. Separating data processing and data storage makes disaster recovery strategies more flexible. Even if the data processing part is damaged, the data storage part can still remain intact, so that services can be quickly restored. The separation of data processing and data storage reduces the dependencies between components and makes the system more modular. This helps to reduce error propagation and improve system stability. The scada host becomes the scada real-time library data host, and the scada microservice does not need to be deployed on the host machine where the scada is located, so that elastic expansion is more flexible.

[0027] In order to provide better security control, the present invention separates data processing and data storage, implements strict security measures for data storage, and implements different security policies for data processing. The separation architecture makes it easier for the system to scale horizontally. Data storage can be expanded independently of data processing to adapt to the growth of data volume, while data processing can independently expand computing resources as needed. The separation architecture allows the system to support multiple data processing requirements at the same time. Different data processing applications can access the same data storage without each application maintaining its own copy of the data.

[0028] The present invention adopts a cloud-edge fusion architecture, which significantly improves the flexibility and scalability of the real-time monitoring system of the power grid. The system can be flexibly expanded and adjusted according to the actual needs of power grid monitoring to adapt to the ever-changing data processing needs and the growth of data volume. The system adopts a modular design, and each module is responsible for different data processing tasks and can be expanded independently without affecting each other. This design enables the system to respond quickly when faced with a sharp increase in data volume or specific processing requirements, and to enhance the processing capabilities of specific modules in a targeted manner. The system adopts a microservice architecture to split complex business logic into a series of small, independent services. Each microservice can be independently deployed, expanded and maintained, which provides great convenience for the expansion of the system. The system has dynamic resource management capabilities and can automatically adjust resource allocation based on the real-time monitored data flow and processing load. This intelligent resource management strategy ensures that the system can maintain efficient operation under different load conditions.

[0029] like Figure 3The diagram shows the architecture of the data high-speed access system of the present invention. The data access technology solution under containerization realizes the separation of data processing and data storage, thereby improving the data processing efficiency of the power grid monitoring system. The distributed architecture is adopted to disperse data and processing tasks to multiple nodes, thereby achieving load balancing, avoiding single-point overload, and significantly improving the overall data processing capability; the realization of the model synchronization function ensures the data consistency between various microservices, reduces the time required for data synchronization, and speeds up the data processing process.

[0030] (1) First-level cache service architecture

[0031] The first-level cache service has three functions: model synchronization, data access to the first-level cache, and data update to the first-level cache. The model synchronization function is triggered when the first-level cache service is started, the model changes, the table is down, the plug-in deletes the record, and it is triggered periodically. This function obtains the model data and builds the cache by calling the model synchronization service sca_model_sync_server of the second-level cache; when the data processing service accesses the real-time library, it calls the first-level cache service through the dynamic library; when the data is updated, it needs to be disassembled by domain, and then the first-level cache is updated, and time stamp verification is required during the update; for the main container, the first-level cache needs to send the updated data to the data update message channel periodically; for the backup container, after receiving the update message channel sent to the changed data, the corresponding cache needs to be updated according to the time stamp verification result.

[0032] like Figure 4 As shown in the figure, the first-level cache service architecture includes libsca_cache_dll.so and sca_inner_cache_server; libsca_cache_dll.so is a dynamic library loaded by the data processing service, through which the data processing service communicates with the first-level cache service. sca_inner_cache_server includes the following modules: interface module, data storage module, data update upstream module, data update downstream module and model synchronization module.

[0033] The interface module includes data reading interface, data updating interface and data adding and deleting interface. The interface module is the interface exposed to libsca_cache_dll.so. The data reading interface and data updating interface give priority to accessing the data storage module. If the data storage module does not have the model, that is, the page is missing, the model needs to be cached to the data storage module by calling the model synchronization module. The data adding and deleting interface directly calls the secondary cache service to update the real-time library and sends the update notification to the internal model update notification channel. The cache data is updated by the model synchronization module.

[0034] The data storage module is the core module of the first-level cache service architecture, storing the real-time library models and data required by this container. When the service starts, the cache of the data storage module is constructed through the model synchronization module; when the interface module accesses a missing page of the data storage module, the model synchronization module is required to update the cache data of the module; when the model is updated (including through periodic updates, downloading real-time library tables, etc.), the model synchronization module is required to update the cache data of the module; the cache size supports dynamic scaling; periodic inspections are performed to see if the cache data has been updated. When the data has been updated, the cache data is updated and the updated data is pushed to the data update channel (container host); when receiving data from the data update channel, the cache is updated after timestamp and node verification.

[0035] The data update uplink module is used to monitor the data update channel. If there is data update from a non-local end, the data is pushed to the data storage module for processing and caching. The data update downlink module is used to push the updated data to the data update channel when there is data update in the data storage module.

[0036] The model synchronization module is used to construct and update the data storage module cache. When the program starts, the model synchronization module calls the secondary cache service on demand to build the cache of the data storage module; the model synchronization module subscribes to the internal and external model update notification channels, calls the secondary cache service on demand, and builds the cache of the data storage model.

[0037] (2) Secondary Cache Service Architecture

[0038] The secondary cache service functions include data update processing service functions, data snapshot processing service functions and model library change monitoring service functions. The update processing service function is to safely store the updated data of scada in the container. The updated data in the container needs to be time-stamped and then written into the time library; time-stamp verification means that the time of pre-updated data is greater than the time of data stored in the real-time library; for updated data, if the source of the data stored in the real-time library is different, the data needs to be snapshot stored to facilitate problem troubleshooting. The data snapshot processing service function can query abnormal data information of measurement and status value data for problem troubleshooting; it supports snapshot query of measurement and status values, and queries the update time stamp, author and abnormal data information of the recent 5 times of the current value of the real-time library. The model library change monitoring service function receives model synchronization requests from the first-level cache; for the requested table model, it monitors the related down table actions. If there is a down table, the host notifies the first-level cache to perform model update processing.

[0039] like Figure 5As shown, the secondary cache service architecture includes a data update processing service module sca_outer_cache_server, a model library change monitoring service module sca_model_chg_server, a data snapshot processing service module sca_meas_snap_server, and a data snapshot query tool sca_meas_snap_tool.

[0040] The data update processing service module is used to receive and process the scada updated data in the container; receive data update channel data, and directly store non-measurement and status value data; use the time stamp storage module to perform time stamp verification on the received data for measurement and status value data, and store the data after verification; and perform time stamp data storage on the update channel data; perform heterogeneous verification on the received measurement data, and if the verification fails, the data needs to be forwarded to the data snapshot channel; and regularly delete expired measurement data and measurement values ​​in the time stamp data storage.

[0041] The model library change monitoring service module is used to process the model synchronization downlink request of the first-level cache service; monitor the real-time library table information downloaded by related applications, and send a message to the internal model update notification channel if the real-time library table download action occurs.

[0042] The data snapshot processing service module is used for snapshot storage of heterogeneous data; receiving data from the data snapshot channel, storing the snapshot into the warehouse; and circularly deleting expired snapshot data.

[0043] The data snapshot query tool is a snapshot query function for measurement and status value data. For heterogeneous data information, you can directly read the snapshot storage module data. When the snapshot storage module does not find the data information, it proves that the queried measurement point is not heterogeneous data, and it is necessary to call the time stamp data storage module for query. For heterogeneous verification of the received measurement data, if the verification fails, the data snapshot storage module needs to be stored.

[0044] (3) scada host

[0045] The scada master and standby will become data storage hosts instead of data processing masters and standbys. That is, the data of the scada master and standby machines all come from the data processed by the main microservice. The data processed by the standby microservice is only stored in the first-level cache or by-pass, and will not be written to the disk. In this way, there will be no jump problem when switching between the master and standby machines.

[0046] The present invention adopts the optimization of the master-slave mechanism, avoids the jump problem, and improves the stability and fault tolerance of the system. The optimized master-slave mechanism realizes the seamless switching function. When the main service fails, the backup service can quickly take over and continue to provide services, thereby avoiding service interruption and data jump problems. The optimized master-slave mechanism also supports load balancing, and can dynamically adjust the resource allocation of the master-slave service according to the real-time load of the system to optimize the overall performance. The master-slave mechanism supports flexible deployment strategies, and can choose to deploy the master-slave service in different physical locations according to actual needs to improve the geographical redundancy and disaster recovery capabilities of the system. By optimizing the master-slave mechanism, the stability and reliability of the system can be significantly improved without significantly increasing costs, thereby maximizing cost-effectiveness.

[0047] In the method for high-speed data access in real-time monitoring of a power grid described in the present invention, the data collected by the power microservice is stored in a first-level cache service architecture, and then the data is updated to a second-level cache service architecture, and finally stored in a scada host machine; when the data processing service performs real-time library access, the first-level cache service is called by means of a dynamic library.

[0048] The electronic device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for high-speed data access in real-time monitoring of a power grid is implemented.

[0049] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the method for high-speed data access in real-time monitoring of a power grid is implemented.

[0050] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0051] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A high-speed data access system for real-time monitoring of power grid, characterized in that: It includes the scada host, the first-level cache service architecture and the second-level cache service architecture; the data collected by the power microservice is stored in the first-level cache service architecture, and then the data is updated to the second-level cache service architecture, and finally stored in the scada host; when the data processing service performs real-time library access, the first-level cache service is called through the dynamic library.

2. The high-speed data access system for real-time monitoring of a power grid according to claim 1, characterized in that: The first-level cache service architecture includes a data storage module and a model synchronization module. The data storage module is used to store the real-time library model and data of the container. When the service is started, the model synchronization module sends a model synchronization downlink request to call the second-level cache service. The data storage module builds the cache of the data storage module through the model synchronization module. The secondary cache service architecture includes a model library change monitoring service module, which is used to process model synchronization downlink requests of the primary cache service and monitor the download of real-time library table information.

3. The high-speed data access system for real-time monitoring of a power grid according to claim 2, characterized in that: The first-level cache service architecture includes an interface module, which includes a data reading interface and a data updating interface; The data reading interface and the data updating interface give priority to accessing the data storage module. If the data storage module does not have the model, that is, the page is missing, the model is cached to the data storage module by calling the model synchronization module.

4. The high-speed data access system for real-time monitoring of a power grid according to claim 2, characterized in that: The interface module also includes a data addition and deletion interface, which directly calls the secondary cache service to update the real-time library and sends the update notification to the internal model update notification channel.

5. The high-speed data access system for real-time monitoring of power grid according to claim 2, characterized in that: The cache size of the data storage module supports dynamic expansion; the data storage module periodically checks whether the cache data has been updated. When the data is updated, the cache data is updated and the updated data is pushed to the data update channel; when receiving data from the data update channel, the cache is updated after timestamp and node verification.

6. The high-speed data access system for real-time monitoring of a power grid according to claim 5, characterized in that: The secondary cache service architecture includes a data update processing service module, which is used to receive and process the data updated by scada in the container; receive data update channel data, and directly store non-measurement and status value data; use the time stamp storage module to perform time stamp verification on the received data of measurement and status value data, and store the data after verification; and perform time stamp data storage on the update channel data; perform heterogeneous verification on the received measurement data, and if the verification fails, the data needs to be forwarded to the data snapshot channel; Regularly delete expired measurement data and measurement values ​​in the time-stamped data storage.

7. The high-speed data access system for real-time monitoring of a power grid according to claim 6, characterized in that: The secondary cache service architecture includes a data snapshot processing service module, which is used for snapshot storage of heterogeneous data; receiving data snapshot channel data, storing the snapshot into the warehouse; and circularly deleting expired snapshot data.

8. A method for high-speed data access in real-time monitoring of a power grid, characterized in that: The data collected by the power microservice is stored in the first-level cache service architecture, and then the data is updated to the second-level cache service architecture, and finally stored in the scada host; when the data processing service performs real-time library access, the first-level cache service is called through the dynamic library.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the method for high-speed data access in real-time monitoring of a power grid according to claim 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for high-speed data access in real-time monitoring of a power grid according to claim 8 is implemented.

Images