Power grid real-time data elastic capacity expansion system based on mass streaming data

Abstract

The invention discloses a power grid real-time data elastic capacity expansion system based on mass streaming data, relates to the technical field of power grid real-time data storage, and aims to solve the problem that a storage environment is not safe due to the fact that more intelligent elastic capacity expansion storage is not adopted when existing power grid data is stored. Through a real-time capacity monitoring function, the system can know the capacity condition of the storage node in time, so that an intelligent storage scheduling decision is made, when the capacity of the storage node is insufficient, a capacity expansion process can be automatically triggered, the need of manual intervention is reduced, and through dynamic capacity expansion and load balancing, storage resources can be more effectively utilized, and the capacity expansion efficiency is improved. By monitoring and analyzing the power grid data in real time, abnormal conditions in the power grid can be found in time, corresponding measures are taken for prevention and processing, and the storage space and transmission bandwidth requirements of the data can be effectively reduced by adopting a lossy compression mode.

Description

Technical Field

[0001] This invention relates to the field of real-time data elastic expansion technology for power grids, specifically a real-time data elastic expansion system for power grids based on massive streaming data. Background Technology

[0002] Real-time data elastic expansion of the power grid refers to dynamically increasing the capacity of data processing and storage resources during power grid operation based on the increase in data volume and the pressure of data processing, in order to ensure the stable operation and performance of the power grid monitoring, analysis and management system.

[0003] Chinese patent CN111414384A discloses a method and apparatus for processing massive streaming data. It primarily involves encoding, compressing, and storing the massive streaming data into corresponding queues upon acquisition or monitoring, reducing or avoiding the impact of sudden surges in streaming data on the data processing system. This facilitates timely and accurate processing of the streaming data, improving the real-time performance and accuracy of massive streaming data processing, and enhancing the response speed of the data processing system. While this patent solves the data storage problem, the following issues still exist in practical operation:

[0004] 1. No further data processing was performed after the power grid data was collected, which reduced the traceability of the original power grid data.

[0005] 2. The lack of real-time analysis of power grid data and the absence of effective data segmentation resulted in poor data accuracy and scheduling.

[0006] 3. The storage environment for power grid data was not effectively processed, and targeted early warnings were not issued based on the degree of abnormality of the abnormal data, resulting in reduced data storage effectiveness and judgment. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time elastic capacity expansion system for power grids based on massive streaming data. Through real-time capacity monitoring, the system can promptly understand the capacity status of storage nodes, thereby making intelligent storage scheduling decisions. When storage node capacity is insufficient, it can automatically trigger the expansion process, reducing the need for manual intervention. Through dynamic expansion and load balancing, storage resources can be utilized more effectively. By monitoring and analyzing power grid data in real time, abnormal situations in the power grid can be detected in a timely manner, and corresponding measures can be taken for prevention and handling. The use of lossy compression can effectively reduce data storage space and transmission bandwidth requirements, solving problems in existing technologies.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A real-time data elastic expansion system for power grids based on massive streaming data includes:

[0010] The data confirmation processing unit is used for:

[0011] Different data are collected based on different deployment devices. After the data is collected, it is processed. After the data is processed, it is uniquely coded and labeled. The target power grid data is obtained after the unique coding and labeling.

[0012] Dynamic data allocation unit, used for:

[0013] The dynamic data in the target power grid data collection is confirmed, and the load status of each dynamic data is processed. After the load status processing is completed, the load is allocated. After the load allocation is completed, the target dynamic data is obtained.

[0014] Dynamic data distributed stream processing unit, used for:

[0015] The target dynamic data is segmented, and after the data segmentation is completed, the data is processed in parallel. After the parallel processing is completed, the data is integrated, and after the data integration is completed, the standard power grid real-time data is obtained.

[0016] Distributed streaming data elastic storage unit, used for:

[0017] The standard power grid real-time data is compressed and deduplicated, an elastic storage framework is constructed, and the compressed and deduplicated data is expanded and stored according to the constructed elastic storage framework to obtain the standard power grid stored data.

[0018] The intelligent monitoring and early warning unit is used for:

[0019] Anomaly thresholds are set based on historical data and threshold rules. The values ​​of each data point in the standard power grid storage data are confirmed. The confirmed data are compared with the anomaly thresholds to obtain abnormal power grid data. Different levels of anomaly warnings are issued based on the anomaly range of the abnormal power grid data.

[0020] Preferably, the data confirmation processing unit includes:

[0021] The data acquisition and preprocessing module is used for:

[0022] The collected data includes power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data;

[0023] Power load data is collected through load monitoring devices; voltage data is collected through voltage sensors; current data is collected through current sensors; frequency data is collected through frequency sensors; equipment status data is collected through relay protection devices; environmental parameter data is collected through temperature and humidity sensors, smoke detectors, and cameras; and energy consumption data is collected through smart meters.

[0024] The collected data undergoes data preprocessing, which includes data cleaning, data transformation, data validation, data filtering, data augmentation, and time synchronization.

[0025] Data cleaning includes removing invalid data, filling in missing values, and standardizing data formats; data transformation includes data type conversion and data scale conversion; data validation includes data range validation and data logic validation; data filtering is noise filtering; data augmentation includes data synthesis and feature extraction; and time synchronization is timestamp calibration of the acquisition equipment.

[0026] Preferably, the data confirmation processing unit further includes:

[0027] The data collection and labeling module is used for:

[0028] Extract feature data from the preprocessed data;

[0029] Feature data includes timestamps, device identifiers, device monitoring data, data preprocessing data, and data quality index data;

[0030] The feature data is processed using a hash function, and the resulting data is mapped to a fixed-length string.

[0031] Each string is combined with feature data to obtain a unique code.

[0032] The unique code is formatted, and the target power grid data is obtained after the formatting process.

[0033] Preferably, the dynamic data allocation unit is further configured to:

[0034] The dynamic data in the target power grid data collection is confirmed. The dynamic data in the target power grid data collection includes power load data, voltage data, current data, frequency data, equipment status data and energy consumption data after unique coding labels.

[0035] After the dynamic data is confirmed, real-time analysis is performed. The real-time analysis determines the real-time requirements of the data by using the timestamp and update frequency of the dynamic data. The real-time requirements are retrieved from the rule database. If the real-time requirements are not met, the data will be collected and processed again.

[0036] Load analysis is performed on dynamic data that meets real-time requirements. Load analysis involves identifying the load status of dynamic data.

[0037] The identified load status is used to formulate a load balancing strategy. The load balancing strategy is to select a load strategy for dynamic data based on the load status. The load strategies include round-robin allocation, minimum connection number allocation, hash-based allocation, and dynamic adjustment allocation.

[0038] Based on priority rules, the load balancing strategy is formulated and priorities are dynamically allocated.

[0039] After priority allocation is completed, the target dynamic data is obtained.

[0040] Preferably, the dynamic data distributed stream processing unit is further configured to:

[0041] The target dynamic data is fragmented according to the fragmentation requirements, which include timestamp data fragmentation, device data fragmentation, and hash function fragmentation.

[0042] Timestamp data sharding divides the target dynamic data according to timestamps; device data sharding divides the target dynamic data according to devices; hash function sharding uses hash functions to map the target dynamic data to different shards.

[0043] After the target dynamic data is fragmented, an identifier is added to each fragment to obtain the dynamic fragmented data;

[0044] Dynamically sharded data is distributed to multiple processing nodes, and data processing tasks are executed on each processing node. These tasks include data analysis, computation, and transformation.

[0045] The dynamic fragmented data completed by the data processing task is coordinated and synchronized between processing nodes using a distributed coordination framework, which retrieves data from the database.

[0046] The processed data fragments from each processing node are collected, and the collected data are integrated to obtain standard real-time power grid data.

[0047] Preferably, the distributed streaming data elastic storage unit includes:

[0048] The power grid data processing module is used for:

[0049] Standard power grid real-time data is compressed using a lossy compression method;

[0050] After confirming the data compression method, the compression strategy will be confirmed, including data accuracy analysis and confirmation of the data loss range.

[0051] Data accuracy analysis involves refining the numerical data in the standard power grid real-time data to two decimal places; data loss range confirmation involves confirming the maximum acceptable error range when compressing the standard power grid real-time data.

[0052] The standard power grid real-time data is compressed according to the compression method and compression strategy.

[0053] After compression is implemented, the compressed data of the standard power grid real-time data is obtained. The compression effect of the compressed data of the standard power grid real-time data is evaluated. The compression effect evaluation includes compression rate evaluation and data quality evaluation. If the evaluation result is not within the preset range, the data compression is performed again.

[0054] After compression is implemented, the compressed standard real-time power grid data is obtained. The compressed standard real-time power grid data is then deduplicated using streaming deduplication.

[0055] After deduplication, the data obtained is the power grid processing data.

[0056] Preferably, the distributed streaming data elastic storage unit further includes:

[0057] The power grid data expansion and storage module is used for:

[0058] The elastic storage framework is constructed, and its storage architecture includes distributed storage, data sharding, and data replication; the network of the elastic storage framework is a load-balanced network; the data center layout of the elastic storage framework includes multi-region deployment and disaster recovery.

[0059] The storage capacity of each data segment of the power grid processing data is confirmed. The storage capacity of the segment is determined by dividing the data segment into several segments of the same length. The storage capacity of the segment data is confirmed based on the data of the segments.

[0060] Then, the memory is monitored in real time. When the fragmented data is stored on multiple storage nodes in the memory, the fragmented data automatically selects the storage node with a fragmented storage amount that is less than the real-time capacity as the final storage node.

[0061] When the storage capacity of all fragmented data exceeds the storage capacity of the real-time storage nodes, the storage nodes will trigger the expansion condition, and storage nodes will be added again after the expansion condition is triggered.

[0062] At the same time, the data in the original storage nodes is migrated to the newly added storage nodes;

[0063] The power grid processing data is expanded and stored to obtain standard power grid storage data.

[0064] Preferably, the intelligent monitoring and early warning unit includes:

[0065] The abnormal data confirmation module is used for:

[0066] Retrieve historical data from the database and confirm the data range under normal operating conditions. The data range includes the normal fluctuation range of load, voltage, current, frequency, and energy consumption.

[0067] Based on the data range under normal operating conditions, set the threshold for each type of data;

[0068] The standard power grid stored data is compared with the set threshold.

[0069] Based on the comparison results, abnormal power grid data in the standard power grid storage data are identified.

[0070] Preferably, the intelligent monitoring and early warning unit further includes:

[0071] The abnormal data early warning module is used for:

[0072] Confirm the difference between abnormal power grid data and the set threshold;

[0073] The degree of abnormality is determined based on the difference, and the degree of abnormality is divided into level one, level two, and level three.

[0074] Different levels of warnings are issued based on the severity of the anomaly, with warning levels divided into Level 1, Level 2, and Level 3.

[0075] The method for elastic scaling of real-time power grid data based on massive streaming data includes the following steps:

[0076] S1: Collect and process power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data, and assign a unique code to each power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data.

[0077] S2: Perform load analysis and formulate load balancing strategies based on the data collected from the target power grid;

[0078] S3: The data after load analysis and load balancing strategy formulation is segmented and processed to obtain standard real-time power grid data.

[0079] S4: Compress and deduplicate the real-time data of the standard power grid. After compression and deduplication, select storage nodes for storage. If the real-time capacity of the storage nodes is insufficient, automatically expand the capacity. After storage, the standard power grid stored data is obtained.

[0080] S5: Perform data anomaly investigation and anomaly warning on standard power grid stored data.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0082] 1. The real-time data elastic expansion system for power grids based on massive streaming data provided by this invention uses timestamp data sharding to facilitate data archiving and cleaning, improving the efficiency of data management. Device data sharding also helps to realize data access control, ensuring data security and privacy. By using hash functions to map target dynamic data to different shards, the uniform distribution of data can be achieved, avoiding the problems of data hotspots and load imbalance.

[0083] 2. The real-time data elastic expansion system for power grids based on massive streaming data provided by this invention can effectively reduce the data storage space and transmission bandwidth requirements by using lossy compression. Data accuracy analysis ensures that the numerical data is accurate to two decimal places, and the confirmation of the data loss range clarifies the maximum acceptable error range during data compression, thereby ensuring the quality controllability of the data during the compression process.

[0084] 3. The real-time data elastic expansion system for power grids based on massive streaming data provided by this invention has a real-time capacity monitoring function that enables the system to understand the capacity status of storage nodes in a timely manner, thereby making intelligent storage scheduling decisions. When the capacity of storage nodes is insufficient, the expansion process can be automatically triggered, reducing the need for manual intervention. Through dynamic expansion and load balancing, storage resources can be utilized more effectively. By monitoring and analyzing power grid data in real time, abnormal situations in the power grid can be detected in a timely manner, and corresponding measures can be taken for prevention and handling. Attached Figure Description

[0085] Figure 1 This is a schematic diagram of the real-time power grid data elastic expansion unit of the present invention;

[0086] Figure 2 This is a schematic diagram of the real-time data elastic expansion process of the power grid according to the present invention. Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] To address the issue of reduced traceability of raw power grid data due to the lack of further data processing after collection in existing technologies, please refer to [link / reference needed]. Figure 1 and Figure 2 This embodiment provides the following technical solution:

[0089] A real-time data elastic expansion system for power grids based on massive streaming data includes:

[0090] The data confirmation processing unit is used for:

[0091] Different data are collected based on different deployment devices. After the data is collected, it is processed. After the data is processed, it is uniquely coded and labeled. The target power grid data is obtained after the unique coding and labeling.

[0092] Dynamic data allocation unit, used for:

[0093] The dynamic data in the target power grid data collection is confirmed, and the load status of each dynamic data is processed. After the load status processing is completed, the load is allocated. After the load allocation is completed, the target dynamic data is obtained.

[0094] Dynamic data distributed stream processing unit, used for:

[0095] The target dynamic data is segmented, and after the data segmentation is completed, the data is processed in parallel. After the parallel processing is completed, the data is integrated, and after the data integration is completed, the standard power grid real-time data is obtained.

[0096] Distributed streaming data elastic storage unit, used for:

[0097] The standard power grid real-time data is compressed and deduplicated, an elastic storage framework is constructed, and the compressed and deduplicated data is expanded and stored according to the constructed elastic storage framework to obtain the standard power grid stored data.

[0098] The intelligent monitoring and early warning unit is used for:

[0099] Anomaly thresholds are set based on historical data and threshold rules. The values ​​of each data point in the standard power grid storage data are confirmed. The confirmed data are compared with the anomaly thresholds to obtain abnormal power grid data. Different levels of anomaly warnings are issued based on the anomaly range of the abnormal power grid data.

[0100] Specifically, the data confirmation processing unit helps optimize the performance of the real-time data elastic expansion system of the power grid; the dynamic data allocation unit prioritizes dynamic data according to priority rules, ensuring that critical data is processed first; the dynamic data distributed stream processing unit adapts to the ever-increasing demand for real-time data of the power grid through intelligent scheduling and elastic expansion technology; the distributed stream data elastic storage unit can automatically trigger expansion conditions and add new storage nodes, thereby achieving seamless capacity expansion; and the intelligent monitoring and early warning unit can promptly detect abnormal situations in the power grid and take corresponding measures for prevention and handling.

[0101] The data confirmation processing unit includes:

[0102] The data acquisition and preprocessing module is used for:

[0103] The collected data includes power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data;

[0104] Power load data is collected through load monitoring devices; voltage data is collected through voltage sensors; current data is collected through current sensors; frequency data is collected through frequency sensors; equipment status data is collected through relay protection devices; environmental parameter data is collected through temperature and humidity sensors, smoke detectors, and cameras; and energy consumption data is collected through smart meters.

[0105] The collected data undergoes data preprocessing, which includes data cleaning, data transformation, data validation, data filtering, data augmentation, and time synchronization.

[0106] Data cleaning includes removing invalid data, filling in missing values, and standardizing data formats; data transformation includes data type conversion and data scale conversion; data validation includes data range validation and data logic validation; data filtering is noise filtering; data augmentation includes data synthesis and feature extraction; and time synchronization is timestamp calibration of the acquisition equipment.

[0107] Specifically, by using different types of sensors and monitoring devices, the diversity and integrity of data are ensured. The use of specialized sensors and monitoring devices for data acquisition, such as load monitoring devices, voltage sensors, and current sensors, improves data accuracy. Data cleaning steps in data preprocessing, including removing invalid data, filling in missing values, and standardizing data formats, further ensure data accuracy and consistency. The data preprocessing process includes multiple stages such as data cleaning, data transformation, data verification, data filtering, data augmentation, and time synchronization. It can efficiently process massive amounts of streaming data, possessing high data throughput and processing capabilities. It can adapt to the high-frequency changes and large-scale growth of real-time power grid data and can easily expand the data acquisition range and preprocessing process to adapt to future power grid development and changes.

[0108] The data collection and labeling module is used for:

[0109] Extract feature data from the preprocessed data;

[0110] Feature data includes timestamps, device identifiers, device monitoring data, data preprocessing data, and data quality index data;

[0111] The feature data is processed using a hash function, and the resulting data is mapped to a fixed-length string.

[0112] Each string is combined with feature data to obtain a unique code.

[0113] The unique code is formatted, and the target power grid data is obtained after the formatting process.

[0114] Specifically, by extracting key feature data, data redundancy is reduced, making subsequent processing and analysis more efficient. By combining each string with feature data, a unique code is obtained, ensuring data uniqueness. This helps to quickly locate specific data in massive datasets, improving data retrieval efficiency. The application of hash functions increases data confidentiality to a certain extent. Due to the one-way and irreversible nature of hash functions, even if data is leaked, attackers find it difficult to recover the original data from the hash value. In the real-time data elastic expansion system of the power grid, data processing and analysis capabilities need to continuously improve with the increase in data volume. This solution reduces the complexity of data processing by extracting key feature data and performing hash processing, making the system easier to achieve elastic expansion. By reducing redundant data and improving data processing efficiency, this solution helps optimize the performance of the real-time data elastic expansion system of the power grid. This includes improving system response speed, reducing system latency, and reducing system resource consumption.

[0115] To address the issues of poor data accuracy and scheduling caused by the lack of real-time analysis and effective data segmentation in existing technologies, please refer to [link to relevant documentation]. Figure 1 and Figure 2 This embodiment provides the following technical solution:

[0116] The dynamic data allocation unit is also used for:

[0117] The dynamic data in the target power grid data collection is confirmed. The dynamic data in the target power grid data collection includes power load data, voltage data, current data, frequency data, equipment status data and energy consumption data after unique coding labels.

[0118] After the dynamic data is confirmed, real-time analysis is performed. The real-time analysis determines the real-time requirements of the data by using the timestamp and update frequency of the dynamic data. The real-time requirements are retrieved from the rule database. If the real-time requirements are not met, the data will be collected and processed again.

[0119] Load analysis is performed on dynamic data that meets real-time requirements. Load analysis involves identifying the load status of dynamic data.

[0120] The identified load status is used to formulate a load balancing strategy. The load balancing strategy is to select a load strategy for dynamic data based on the load status. The load strategies include round-robin allocation, minimum connection number allocation, hash-based allocation, and dynamic adjustment allocation.

[0121] Based on priority rules, the load balancing strategy is formulated and priorities are dynamically allocated.

[0122] After priority allocation is completed, the target dynamic data is obtained.

[0123] Specifically, by verifying dynamic data, including power load data, voltage data, current data, frequency data, equipment status data, and energy consumption data, the accuracy and reliability of the data are ensured. Real-time analysis ensures that the data meets time-sensitive requirements. The real-time nature of the data is determined by timestamps and update frequency, and compared with the real-time requirements in the rule database. Data that does not meet the requirements is re-collected and processed. This helps the system obtain the latest data in a timely manner, improving response speed. Load analysis and the formulation of load balancing strategies help optimize the resource allocation of the power grid system. Selecting appropriate load strategies based on load status (such as round-robin allocation, minimum connection allocation, hash-based allocation, and dynamic adjustment allocation) ensures balanced utilization of system resources, avoids overload of certain nodes or equipment, and improves the overall performance and stability of the system. Prioritizing dynamic data according to priority rules ensures that critical data is processed first. This helps the power grid system prioritize processing data that is crucial to system stability and security when faced with massive amounts of data, thereby improving the system's emergency response capabilities and overall performance. Through steps such as dynamic data verification, real-time analysis, load analysis, and the formulation of load balancing strategies, the system can flexibly cope with the growth of data volume. The target dynamic data processed by this solution can help the power grid system make decisions more quickly, improving decision-making efficiency and accuracy.

[0124] The dynamic data distributed stream processing unit is also used for:

[0125] The target dynamic data is fragmented according to the fragmentation requirements, which include timestamp data fragmentation, device data fragmentation, and hash function fragmentation.

[0126] Timestamp data sharding divides the target dynamic data according to timestamps; device data sharding divides the target dynamic data according to devices; hash function sharding uses hash functions to map the target dynamic data to different shards.

[0127] After the target dynamic data is fragmented, an identifier is added to each fragment to obtain the dynamic fragmented data;

[0128] Dynamically sharded data is distributed to multiple processing nodes, and data processing tasks are executed on each processing node. These tasks include data analysis, computation, and transformation.

[0129] The dynamic fragmented data completed by the data processing task is coordinated and synchronized between processing nodes using a distributed coordination framework, which retrieves data from the database.

[0130] The processed data fragments from each processing node are collected, and the collected data are integrated to obtain standard real-time power grid data.

[0131] Specifically, by partitioning target dynamic data according to timestamps, time-series data management can be easily achieved, which is beneficial for time series analysis, such as trend prediction and anomaly detection. Simultaneously, timestamp data sharding facilitates data archiving and cleanup, improving data management efficiency. Partitioning by device allows for more centralized device-related data, facilitating tasks such as device status monitoring and fault diagnosis. Furthermore, device data sharding helps in data access control, ensuring data security and privacy. Using hash functions to map target dynamic data to different shards achieves even data distribution, avoiding data hotspots and load imbalances. Hash sharding is simple, efficient, and highly scalable, suitable for handling massive data storage and high-concurrency access scenarios. After sharding, an identifier is added to each shard, facilitating data tracking and management. This sharding method also enables elastic system expansion. As data volume increases, the number of shards can be increased to expand the system's storage and processing capacity. Distributing dynamically sharded data across multiple processing nodes and executing data processing tasks on each node enables parallel data processing, improving system efficiency. Furthermore, when a processing node experiences excessive load, some data can be sharded and migrated to other nodes to achieve load balancing and ensure stable system operation. Utilizing a distributed coordination framework for coordination and synchronization among processing nodes ensures data consistency and task collaboration. This coordination mechanism is crucial for achieving high availability and fault tolerance. Through elastic scaling and dynamic data sharding, the system's computing and storage resources can be fully utilized, avoiding resource idleness and waste. This helps reduce system operating costs and improve resource utilization. Intelligent scheduling and elastic scaling technologies can adapt to the ever-increasing demand for real-time power grid data.

[0132] To address the issues in existing technologies where there is a lack of effective processing of the power grid data storage environment and a lack of targeted early warning based on the severity of abnormal data, leading to reduced data storage effectiveness and judgment, please refer to [link to relevant documentation]. Figure 1and Figure 2 This embodiment provides the following technical solution:

[0133] Distributed streaming data elastic storage unit, including:

[0134] The power grid data processing module is used for:

[0135] Standard power grid real-time data is compressed using a lossy compression method;

[0136] After confirming the data compression method, the compression strategy will be confirmed, including data accuracy analysis and confirmation of the data loss range.

[0137] Data accuracy analysis involves refining the numerical data in the standard power grid real-time data to two decimal places; data loss range confirmation involves confirming the maximum acceptable error range when compressing the standard power grid real-time data.

[0138] The standard power grid real-time data is compressed according to the compression method and compression strategy.

[0139] After compression is implemented, the compressed data of the standard power grid real-time data is obtained. The compression effect of the compressed data of the standard power grid real-time data is evaluated. The compression effect evaluation includes compression rate evaluation and data quality evaluation. If the evaluation result is not within the preset range, the data compression is performed again.

[0140] After compression is implemented, the compressed standard real-time power grid data is obtained. The compressed standard real-time power grid data is then deduplicated using streaming deduplication.

[0141] After deduplication, the data obtained is the power grid processing data.

[0142] Specifically, using lossy compression can effectively reduce data storage space and transmission bandwidth requirements, which is crucial for processing massive amounts of real-time power grid data. By confirming the compression strategy (including data accuracy analysis and data loss range confirmation), higher compression efficiency can be achieved while ensuring data quality. Data accuracy analysis ensures that numerical data is accurate to two decimal places, which is crucial for the accurate analysis of power grid data. Data loss range confirmation clarifies the maximum acceptable error range during data compression, thereby ensuring the controllability of data quality during the compression process. Compression effect evaluation includes compression rate evaluation and data quality evaluation, which helps to promptly identify and resolve potential problems during the compression process. If the evaluation results are not within the preset range, data compression is performed again. This feedback mechanism ensures the stability and reliability of the compression results. Data deduplication is performed using streaming deduplication, which means that the system can process and analyze streaming data in real time. This is particularly important for real-time power grid data monitoring and early warning systems. Streaming data processing capabilities improve the system's response speed and real-time performance, enabling the system to adapt to changes in power grid data more quickly.

[0143] The power grid data expansion and storage module is used for:

[0144] The elastic storage framework is constructed, and its storage architecture includes distributed storage, data sharding, and data replication; the network of the elastic storage framework is a load-balanced network; the data center layout of the elastic storage framework includes multi-region deployment and disaster recovery.

[0145] The storage capacity of each data segment of the power grid processing data is confirmed. The storage capacity of the segment is determined by dividing the data segment into several segments of the same length. The storage capacity of the segment data is confirmed based on the data of the segments.

[0146] Then, the memory is monitored in real time. When the fragmented data is stored on multiple storage nodes in the memory, the fragmented data automatically selects the storage node with a fragmented storage amount that is less than the real-time capacity as the final storage node.

[0147] When the storage capacity of all fragmented data exceeds the storage capacity of the real-time storage nodes, the storage nodes will trigger the expansion condition, and storage nodes will be added again after the expansion condition is triggered.

[0148] At the same time, the data in the original storage nodes is migrated to the newly added storage nodes;

[0149] The power grid processing data is expanded and stored to obtain standard power grid storage data.

[0150] Specifically, the system dynamically adds storage nodes to cope with data growth, ensuring it can handle the ever-increasing real-time power grid data. When the storage demand for fragmented data exceeds the capacity of the current storage nodes, the system automatically triggers expansion conditions and adds new storage nodes, achieving seamless capacity expansion. Utilizing a load balancing network ensures data is evenly distributed across multiple storage nodes, avoiding single-point overload issues. Fragmented data is automatically stored on storage nodes with fragmented storage capacity less than the real-time capacity, further balancing the load on storage nodes. Data sharding and replication technologies ensure redundant data storage, improving data reliability and availability. Multi-region deployment and disaster recovery strategies further enhance the system's fault tolerance; even if some storage nodes fail, data can be quickly recovered from other nodes. Real-time capacity monitoring allows the system to understand the capacity status of storage nodes in a timely manner, enabling intelligent storage scheduling decisions. When storage node capacity is insufficient, the system automatically triggers expansion processes, reducing the need for manual intervention. Through dynamic expansion and load balancing, the system can utilize storage resources more effectively, avoiding unnecessary resource waste.

[0151] The intelligent monitoring and early warning unit includes:

[0152] The abnormal data confirmation module is used for:

[0153] Retrieve historical data from the database and confirm the data range under normal operating conditions. The data range includes the normal fluctuation range of load, voltage, current, frequency, and energy consumption.

[0154] Based on the data range under normal operating conditions, set the threshold for each type of data;

[0155] The standard power grid stored data is compared with the set threshold.

[0156] Based on the comparison results, abnormal power grid data in the standard power grid storage data are identified.

[0157] Specifically, through a streaming big data architecture, the system can retrieve historical data from the database in real time and quickly confirm the data range under normal operating conditions. It can also compare the data stored in the standard power grid with the set thresholds in real time, promptly detect abnormal data, and improve the timeliness of data processing. Based on the statistical analysis of historical data, it can accurately set the normal fluctuation range of data such as load, voltage, current, frequency, and energy consumption. According to different power grid operating conditions and needs, the thresholds of various types of data can be flexibly adjusted. By setting reasonable thresholds, the system can filter out important data and information, avoiding excessive processing of unimportant or less influential data. Real-time comparison and anomaly detection can promptly detect potential problems in the power grid, reducing the time for fault occurrence and recovery. By comparing the data stored in the standard power grid with the set thresholds in real time, the system can achieve intelligent monitoring of the power grid operating status.

[0158] The abnormal data early warning module is used for:

[0159] Confirm the difference between abnormal power grid data and the set threshold;

[0160] The degree of abnormality is determined based on the difference, and the degree of abnormality is divided into level one, level two, and level three.

[0161] Different levels of warnings are issued based on the severity of the anomaly, with warning levels divided into Level 1, Level 2, and Level 3.

[0162] Specifically, the system can process and analyze power grid data in real time. Once abnormal data is detected, it is immediately compared with set thresholds to quickly determine the severity of the anomaly and trigger an early warning. This real-time capability is crucial for the stable operation of the power grid, helping to promptly identify problems and take measures to prevent escalation. Based on massive streaming data, the power grid real-time data elastic expansion system can continuously receive and process power grid data without waiting for a certain amount of data to accumulate. This significantly improves data processing efficiency, making anomaly detection more timely and accurate. By setting different levels of thresholds, the system can accurately determine the severity of power grid data anomalies. The classification of Level 1, Level 2, and Level 3 anomalies helps to categorize and handle different types of anomalies, improving the targeting and accuracy of early warnings. Different levels of early warning are triggered based on the severity of the anomaly. The classification of Level 1, Level 2, and Level 3 early warnings allows power grid management departments to take corresponding countermeasures based on different levels of warnings, thus more effectively responding to power grid faults and dynamically adjusting the allocation and scheduling of power grid resources based on real-time changes in power grid data. When abnormal data is detected, an early warning can be triggered immediately, alerting power grid management to pay attention and take appropriate measures to optimize the power grid's operating status. By monitoring and analyzing power grid data in real time, this solution can promptly identify anomalies in the power grid and take corresponding preventative and handling measures. This helps reduce the probability of power grid failures and improves the reliability of power grid operation.

[0163] The method for elastic scaling of real-time power grid data based on massive streaming data includes the following steps:

[0164] S1: Collect and process power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data, and assign a unique code to each power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data, and energy consumption data.

[0165] Assigning a unique code to each data item helps with data tracking, management, and analysis, improving the efficiency and accuracy of data processing.

[0166] S2: Perform load analysis and formulate load balancing strategies based on the data collected from the target power grid;

[0167] Among them, the load balancing strategy can be dynamically adjusted according to the actual situation of the power grid to adapt to changes in the power grid load and ensure the reliable operation of the power grid.

[0168] S3: The data after load analysis and load balancing strategy formulation is segmented and processed to obtain standard real-time power grid data.

[0169] Among these, the fragmented data is easier to manage and maintain, such as data backup, recovery, and migration.

[0170] S4: Compress and deduplicate the real-time data of the standard power grid. After compression and deduplication, select storage nodes for storage. If the real-time capacity of the storage nodes is insufficient, automatically expand the capacity. After storage, the standard power grid stored data is obtained.

[0171] When the real-time capacity of a storage node is insufficient, automatic expansion can ensure continuous data storage and processing, avoiding data loss or processing delays.

[0172] S5: Perform data anomaly investigation and anomaly warning on standard power grid stored data;

[0173] By conducting anomaly checks on standard power grid stored data, outliers or erroneous data can be detected and processed in a timely manner, thereby improving the accuracy and reliability of the data.

[0174] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0175] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A real-time data elastic expansion system for power grids based on massive streaming data, characterized in that: include: The data confirmation processing unit is used to: Different data are collected according to different deployed equipment, and data processing is performed after data collection is completed. After data processing is completed, unique coding and labeling are performed, and target power grid collection data is obtained after unique coding and labeling; Dynamic data allocation unit for: The dynamic data in the target power grid collected data is confirmed, and the load condition of each dynamic data is processed. After the load condition processing is completed, the load is distributed. After the load distribution is completed, the target dynamic data is obtained; Dynamic data distributed stream processing unit for: The target dynamic data is segmented, and after the segmentation, the data is processed in parallel, and after the parallel processing, the data is integrated, and after the data integration, the standard power grid real-time data is obtained; Distributed elastic storage unit for streaming data, used for: The real-time data of the standard power grid is compressed and deduplicated, an elastic storage framework is constructed, and the compressed and deduplicated data is expanded and stored according to the constructed elastic storage framework, and the standard power grid storage data is obtained after the expanded storage; Intelligent monitoring and early warning unit, used for: The abnormal threshold is set according to historical data and threshold rules, the value of each data in the standard power grid storage data is confirmed, and the confirmed data is compared with the abnormal threshold. After the abnormal comparison, abnormal power grid data is obtained, and different degrees of abnormal warning are issued according to the abnormal range of the abnormal power grid data.

2. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 1 is characterized in that: The data confirmation processing unit includes: Data acquisition preprocessing module, used for: The collected data include power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data and energy consumption data; Power load data is collected through load monitoring devices; voltage data is collected through voltage sensors; current data is collected through current sensors; frequency data is collected through frequency sensors; equipment status data is collected through relay protection devices; environmental parameter data is collected through temperature and humidity sensors, smoke detectors and cameras; energy consumption data is collected through smart meters; Preprocess the collected data, including data cleaning, data conversion, data verification, data filtering, data enhancement and time synchronization; Data cleaning includes removing invalid data, filling missing values ​​and standardizing data formats; data conversion includes data type conversion and data scale conversion; data validation includes data range validation and data logic validation; data filtering is noise filtering; data enhancement includes data synthesis and feature extraction; time synchronization is the timestamp calibration of the acquisition device.

3. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 2 is characterized in that: The data confirmation processing unit further includes: The data collection labeling module is used to: Extracting feature data from the data preprocessed; Feature data includes timestamp, device identifier, device monitoring data, data preprocessing data, and data quality indicator data; The feature data is processed using a hash function, and different data obtained after feature processing are mapped to a string of fixed length; Combine each character string with the characteristic data to obtain a unique code. The unique code is formatted, and the target power grid collection data is obtained after the formatting.

4. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 3 is characterized in that: The dynamic data allocation unit is further used for: Confirm the dynamic data in the target power grid collected data, which are the power load data, voltage data, current data, frequency data, equipment status data and energy consumption data after unique coding; After the dynamic data is confirmed, real-time analysis is performed. The real-time analysis determines the real-time requirements of the data through the timestamp and update frequency of the dynamic data. The real-time requirements are retrieved from the rule database. If the real-time requirements are not met, the data will be collected and processed again. Perform load analysis on dynamic data that meets the real-time requirements. Load analysis is to identify the load status of dynamic data. Formulate a load balancing strategy for the identified load status. The load balancing strategy is to select a load strategy for dynamic data according to the load status. The load strategies include polling allocation, minimum connection number allocation, hash-based allocation, and dynamic adjustment allocation. According to the priority rules, the load balancing strategy is formulated and priority is assigned dynamically; After the priority allocation is completed, the target dynamic data is obtained.

5. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 4 is characterized in that: The dynamic data distributed stream processing unit is further used for: Shard the target dynamic data according to the sharding requirements, which include timestamp data sharding, device data sharding, and hash function sharding; Timestamp data sharding is to divide the target dynamic data according to timestamps; Device data sharding is to divide the target dynamic data according to the device; Hash function sharding uses hash functions to map target dynamic data to different shards; After the target dynamic data is sharded, an identifier is added to each shard to obtain dynamic shard data; Distribute dynamically sharded data to multiple processing nodes and execute data processing tasks on each processing node. Data processing tasks include data analysis, calculation, and conversion. The distributed coordination framework is used to coordinate and synchronize the dynamic shard data after the data processing task is completed, and the distributed coordination framework is retrieved from the database; The data slicing results processed on each processing node are collected, and the collected data results are integrated to obtain the standard power grid real-time data.

6. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 5 is characterized in that: The distributed stream data elastic storage unit includes: Power grid data processing module for: Compress the standard power grid real-time data using a lossy compression method; After the data compression method is confirmed, the compression strategy will be confirmed, which includes data accuracy analysis and data loss range confirmation; Data accuracy analysis is to accurately calculate the numerical data in the standard power grid real-time data to two decimal places; data loss range confirmation is to confirm the maximum acceptable error range when the standard power grid real-time data is compressed; Compressing and implementing standard power grid real-time data according to compression methods and compression strategies; After the compression is implemented, the compressed data of the standard power grid real-time data is obtained, and the compression effect of the compressed data of the standard power grid real-time data is evaluated. The compression effect evaluation includes compression rate evaluation and data quality evaluation. If the evaluation result is not within the preset range, the data is compressed again; After the compression is implemented, the compressed standard power grid real-time data is obtained, and the compressed standard power grid real-time data is deduplicated, and the data deduplication is stream deduplication; After data deduplication, power grid processing data is obtained.

7. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 6 is characterized in that: The distributed stream data elastic storage unit further includes: Power grid data expansion storage module, used for: The elastic storage framework is constructed. The storage architecture of the elastic storage framework includes distributed storage, data sharding and data replication. The network of the elastic storage framework is a load balancing network. The data center layout of the elastic storage framework includes multi-region deployment and disaster recovery. Confirm the shard storage capacity of each data shard of the power grid processing data. The shard storage capacity is obtained by dividing the shard data into a number of segments of the same length and confirming the shard storage capacity of the shard data according to the data of the segments; Then the storage is monitored for real-time capacity. When the shard data is stored on multiple storage nodes in the storage, the shard data automatically selects the storage node with a shard storage capacity less than the real-time capacity as the final storage node. When the shard storage capacity of the shard data is greater than the real-time capacity of the storage node, the storage node triggers the expansion condition, and after the expansion condition is triggered, the storage node is added again; At the same time, migrate the data in the original storage node to the newly added storage node; The power grid processing data are all expanded and stored to obtain standard power grid storage data.

8. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 7, characterized in that: The intelligent monitoring and early warning unit comprises: Abnormal data confirmation module, used for: Retrieve historical data from the database and confirm the data range under normal operating conditions, including the normal fluctuation range of load, voltage, current, frequency and energy consumption; According to the data range under normal operating conditions, the threshold value of each type of data is set; Perform threshold comparison between the standard power grid storage data and the set threshold; The abnormal power grid data in the standard power grid storage data is judged according to the comparison results.

9. The real-time data elastic expansion system for power grid based on massive streaming data according to claim 8, characterized in that: The intelligent monitoring and early warning unit also includes: Abnormal data warning module, used for: Confirm the difference between abnormal power grid data and the set threshold value; The degree of abnormality is judged based on the difference, and the degree of abnormality is divided into first-level abnormality, second-level abnormality and third-level abnormality; Then, different levels of warnings are issued according to the degree of abnormality, which are divided into level one, level two and level three.

10. A method for elastically expanding real-time data in a power grid based on massive streaming data, applied in a system for elastically expanding real-time data in a power grid based on massive streaming data as claimed in claim 9, characterized in that: The steps include: S1: Collect and process power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data and energy consumption data, and uniquely encode and label each power load data, voltage data, current data, frequency data, equipment status data, environmental parameter data and energy consumption data; S2: Collect data from the target power grid for load analysis and load balancing strategy formulation; S3: Slice the data of load analysis and load balancing strategy formulation to obtain standard power grid real-time data; S4: compress and deduplicate the real-time data of the standard power grid, select a storage node for storage after the data compression and deduplication, and automatically expand the capacity of the storage node when the real-time capacity is insufficient, and obtain the standard power grid storage data after storage; S5: Check the data anomaly and issue anomaly warning for the standard power grid storage data.

Images