Electric energy acquisition data intelligent transmission system based on distributed cooperation and high concurrency mechanism

Abstract

The invention relates to the technical field of electric energy data processing and transmission, in particular to an electric energy acquisition data intelligent transmission system based on a distributed cooperation and high concurrency mechanism. Comprising a multi-layer distributed node architecture module, a high-concurrency transmission module, a data processing and optimizing module and a deep learning module which are connected in sequence, the multi-layer distributed node architecture module adopts a dynamic load balancing algorithm and a data redundancy and recovery mechanism, and the multi-layer distributed node architecture module comprises a data acquisition layer, a data relay layer and a data convergence layer; the high-concurrency transmission module comprises a multi-thread concurrency transmission model, a self-adaptive flow control algorithm and an asynchronous event-driven transmission unit, and dynamically adjusts the number of threads and a sending window according to a network state; and the data processing and optimizing module performs preprocessing and feature extraction on the electric energy data at the data acquisition layer and the transmission node. Data interaction and cooperative work among links of the power system are promoted, and efficient sharing and deep application of power data are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of electric energy data processing and transmission, and in particular to an intelligent transmission system for electric energy acquisition data based on distributed collaboration and high concurrency mechanism. Background Art

[0002] With the continuous growth of global energy demand and the in-depth advancement of smart grid construction, the scale and complexity of power systems are constantly increasing. In modern power systems, a large number of power collection points are widely distributed, and the amount of data generated is growing exponentially. The traditional centralized power collection and data transmission architecture has been unable to adapt to this development trend and has exposed many drawbacks.

[0003] From the perspective of data transmission, the centralized architecture has a high risk of single point failure. Once the central node fails, the entire data transmission system will be paralyzed, and a large amount of power data cannot be collected and transmitted normally, which seriously affects the operation monitoring and management decision-making of the power system. At the same time, in the face of high concurrent data transmission needs, its processing efficiency is low, and data congestion is prone to occur, resulting in a significant increase in data transmission delays, which cannot meet the power system's strict requirements for data real-time performance, affecting the development of key businesses such as power dispatching and fault diagnosis.

[0004] In a complex power grid environment, network fluctuations are frequent and node load imbalance is common. Existing technologies lack effective means to deal with these problems and it is difficult to ensure the stability and accuracy of power data transmission. For example, when the network is congested or interfered, the probability of data loss and errors increases significantly, reducing data quality and posing great challenges to the reliable operation of the power system.

[0005] In addition, with the development of intelligent power systems, the requirements for the processing and analysis of power data are getting higher and higher. The data processing method under the traditional architecture is simple and extensive, which cannot fully tap the value of data and cannot meet the power system's needs for refined management and intelligent decision-making. Therefore, it is urgent to develop an innovative intelligent transmission system for power collection data to break through the bottleneck of existing technologies and promote the sustainable development of the power industry. Summary of the invention

[0006] The purpose of this invention is to construct an advanced intelligent transmission system for electric energy collection data based on distributed collaboration and high concurrency mechanism. By integrating cutting-edge technologies and innovative algorithms, the key problems in the existing power data transmission and processing process are solved, and an intelligent transmission system for electric energy collection data based on distributed collaboration and high concurrency mechanism is proposed.

[0007] The technical solution of the present invention is: an intelligent transmission system for electric energy acquisition data based on distributed collaboration and high concurrency mechanism, comprising a multi-layer distributed node architecture module, a high concurrency transmission module, a data processing and optimization module and a deep learning module connected in sequence, wherein:

[0008] The multi-layer distributed node architecture module adopts a dynamic load balancing algorithm and a data redundancy and recovery mechanism. The multi-layer distributed node architecture module includes a data acquisition layer, a data relay layer and a data aggregation layer. The data acquisition layer includes multiple distributed power acquisition terminals, which are divided into multiple subnets based on geographical location or power grid area. The nodes in the subnet communicate using a multi-hop wireless network protocol to form a local mesh network. The data relay layer deploys multiple high-performance relay nodes for forwarding data and optimizing routing, and dynamically adjusts the data flow direction according to the network status. The data aggregation layer is composed of a central server cluster to aggregate, store, analyze and process data.

[0009] High-concurrency transmission module, including multi-threaded concurrent transmission model, adaptive flow control algorithm and asynchronous event-driven transmission unit, dynamically adjusts the number of threads and sending window according to network status;

[0010] The data processing and optimization module pre-processes and extracts features of the electric energy data at the data collection layer and transmission nodes, and uses a reinforcement learning algorithm to optimize the transmission path, and sets a cache mechanism at the relay nodes and aggregation nodes corresponding to the data relay layer and the data aggregation layer to optimize the data;

[0011] The deep learning module uses convolutional neural networks to model power collection data and uses generative adversarial networks or variational autoencoders to repair abnormal data.

[0012] Optionally, the dynamic load balancing algorithm uses an improved weighted round-robin algorithm to balance node loads, i Assign dynamic weight w i (t), considering the CPU usage u of the node i (t), memory idle rate m i (t), network bandwidth utilization b i (t) and historical task processing success rate s i的 Factors, weight calculation formula is:

[0013]

[0014] Among them, α, β, γ, § are weight coefficients, and α+β+γ+§=1.

[0015] Optionally, the data redundancy and recovery mechanism uses erasure coding technology to perform data redundant storage and transmission, and divides the original power collection data into k data blocks D = {d1, d2, ..., d k}, generate n redundant blocks R = {r1, r2, ..., r n}, forming a (k+n) redundancy scheme.

[0016] Optionally, the multi-threaded concurrent transmission model creates multiple threads T = {t1, t2, ..., t p}Concurrent transmission, introducing thread priority queue and dynamic thread pool management technology, according to the data priority P d Dynamically adjust the number of threads p and thread task allocation according to network load conditions;

[0017] The number of priority threads is adjusted according to the network status, and the thread task allocation formula is as follows:

[0018]

[0019] where t alloc (i) is thread t i The amount of tasks assigned, P d (i) is the priority of data block i, N task is the total task volume, P d (j) represents the priority of data block j. The larger the priority value, the more important or urgent the data block is.

[0020] Optionally, the adaptive flow control algorithm is used to adaptively control the flow of the sending window adjustment algorithm. The part responsible for data transmission in the high-concurrency transmission module is the sending end. In the transmission from the data relay layer to the data convergence layer, the node of the data convergence layer is the receiving end. The sending end adjusts the receiving window size R according to the feedback of the receiving end. w , round trip delay t RTT and network congestion level C to dynamically adjust the sending window S w With the sending rate Sr, the formula is as follows:

[0021]

[0022] Among them, α, β, γ are adjustment coefficients, and τ is the congestion threshold.

[0023] Optionally, the asynchronous event-driven transmission unit constructs a transmission architecture based on an asynchronous I / O and event-driven model, abstracts data transmission events into independent event processing modules, and executes asynchronously when an event is triggered without blocking the main thread;

[0024] When data is sent, the main thread submits the sending task and continues to process other transactions. The sending module executes the sending operation in the background according to the network status and strategy, and notifies the main thread through the callback function when it is completed or an error occurs.

[0025] Optionally, when the data processing and optimization module preprocesses the electric energy data and extracts features, it first performs a discrete Fourier transform on the electric energy data sequence x(n):

[0026]

[0027] X(k) represents the frequency domain component at frequency k after discrete Fourier transform; N represents the number of sampling points, that is, the length of the electric energy data sequence x(n); k is the frequency index, which is used to specify different frequency points in the frequency domain;

[0028] After obtaining the frequency domain information, the principal component analysis method is used to reduce the dimension and extract the key feature vector F = {f1,f2,..,f m}, reduce the amount of data and retain core information.

[0029] Optionally, in the reinforcement learning algorithm, the agent selects the transmission path action a through the policy network π(s) according to the network state observation value s, and the environment feedback reward r. The deep Q network is used to train the agent, and the loss function is:

[0030]

[0031] where y = r + γmax a′ Q(s′,a′;θ - ) is the target Q value, θ is the network parameter, γ is the discount factor, s′ is the new state after executing action a, a′ is all possible actions in the new state s′, θ - are the parameters of the target network.

[0032] Optionally, the cache mechanism combines the least recently used algorithm with data timeliness management cache to record the access time t for the cache data block Bi. access (i) and update time t update (i), the cache replacement formula is:

[0033] B replace =argmin i (t access (i))∧(t update (i) <T expire )

[0034] T expire It is a time threshold used to measure the timeliness of cached data blocks;

[0035] At the same time, cached data is checked regularly based on data update frequency and importance, and expired data is updated or deleted.

[0036] Optionally, the deep learning module constructs an anomaly detection model based on deep learning. The anomaly detection model based on deep learning identifies data points that deviate from the normal pattern as abnormal data by learning the normal pattern of the data. The training formula of the anomaly detection model is:

[0037]

[0038] where y i For real data, is the model prediction data, N is the number of data samples;

[0039] When the deep learning module repairs abnormal data, it generates a repair value similar to normal data by learning the distribution of the data. The data repair formula is:

[0040]

[0041] Where D normal is normal data, D abnormal For abnormal data, The repaired data.

[0042] Optionally, the deep learning module also includes building a real-time monitoring and dynamic feedback mechanism to monitor the anomaly detection and repair process in real time, and dynamically adjust the parameters of the detection and repair model according to the monitoring results to ensure that the model maintains high performance under different operating conditions. The feedback formula for dynamic adjustment is:

[0043]

[0044] Where θ is the model parameter, η is the learning rate, is the gradient of the loss function.

[0045] Compared with the prior art, the present invention has at least one of the following beneficial technical effects:

[0046] 1. Efficient transmission performance: Through multi-layer distributed node architecture modules and high-concurrency transmission modules, the transmission efficiency of power collection data is significantly improved, the delay is reduced, and the power system's requirements for real-time data are met.

[0047] 2. Enhanced reliability and fault tolerance: Adopt dynamic load balancing and data redundancy technology to effectively deal with node failures and network fluctuations, and ensure data integrity and availability.

[0048] 3. Optimal resource utilization: Adaptive algorithms and intelligent strategies allocate resources reasonably according to data priority and network status, improving system resource utilization and cost-effectiveness.

[0049] 4. Intelligent decision support: The intelligent data processing and optimization module as well as the anomaly detection and repair mechanism provide accurate data support for the operation analysis, fault diagnosis and load forecasting of the power system, and help upgrade the intelligent power management. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0052] like Figure 1 The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism of the present invention is mainly composed of a multi-layer distributed node architecture module, a high concurrency transmission module, a data processing and optimization module and a deep learning module. The multi-layer distributed node architecture module includes a data acquisition layer, a data relay layer and a data aggregation layer.

[0053] Implementation of data collection layer:

[0054] In the urban smart grid, the data collection layer is distributed in various power consumption areas. In residential areas, smart meters are installed in the distribution boxes of each household, and the transformer collection module is deployed in the community distribution room to collect more accurate power parameters. These collection terminals collect power data such as voltage, current, and power through built-in high-precision sensors at certain time intervals (for example, every 15 minutes).

[0055] The data collection terminals communicate with the ZigBee multi-hop wireless network protocol to form a local mesh network. Each data collection terminal is both a data collection device and a network node. When a terminal is blocked from communicating with the upper node, such as when the signal is weakened due to building obstruction, the data can be transmitted through the adjacent terminal via multi-hop routing to ensure the reliability and coverage of data collection. In areas with weak signals, signal strength is enhanced by adding signal relay equipment to ensure stable data transmission.

[0056] Data relay layer implementation:

[0057] The high-performance relay nodes of the data relay layer are set up in key locations such as substations and communication base stations. These nodes are equipped with multi-core processors, large-capacity memory and high-speed network interfaces. Taking the relay node in a substation as an example, it establishes a wireless connection with the terminal of the data collection layer through a directional antenna, and is connected to the central server cluster of the data aggregation layer through optical fiber.

[0058] Relay nodes use a dynamic load balancing algorithm to process and forward data. They monitor their own CPU usage, memory idle rate, network bandwidth utilization and other indicators in real time, and broadcast this information to other nodes every 1 minute. When data arrives, the weight of each node is calculated according to the improved weighted polling algorithm, and the data is distributed to nodes with lighter loads. During peak hours of electricity consumption, network traffic in some areas increases dramatically. For example, when the electricity load in a commercial area increases, the relay nodes automatically adjust the data routing and disperse the data traffic to nodes with lighter loads to avoid network congestion and ensure fast data forwarding.

[0059] The relay node uses an adaptive flow control algorithm to dynamically adjust the sending window and sending rate based on the receiving window size, round-trip delay and network congestion level fed back by the receiving end. The congestion level is evaluated by monitoring the packet loss rate and delay in the network. When network congestion is detected, the sending window is reduced in time, the sending rate is reduced to avoid data loss; when the network condition is good, the sending window is increased to improve transmission efficiency.

[0060] Data aggregation layer implementation:

[0061] The data aggregation layer is composed of a central server cluster, which is set up in the data center of the power company. The cluster adopts a distributed storage architecture and uses the Ceph distributed storage system to store power data in multiple server nodes. Each data block has multiple copies to ensure high availability and redundant backup of the data.

[0062] After the central server cluster receives data from the data relay layer, it first unifies the data format and verifies the integrity. It converts the data format that does not conform to the specification, marks the data that fails the verification and requests retransmission. Then, it classifies and stores the data according to the source, time and other information of the data, which is convenient for subsequent query and analysis.

[0063] For a large amount of historical data, distributed database technology is used for management, such as using HBase database to store structured data and using MapReduce framework to process and analyze data in batches. By analyzing historical data, the value of data is mined to provide decision support for the operation optimization of the power system.

[0064] Intelligent data processing and optimization implementation of data processing and optimization module:

[0065] At the collection terminal and transmission node, a method combining discrete Fourier transform (DFT) and principal component analysis (PCA) is used to preprocess and extract features of power data. After collecting data, the collection terminal immediately performs DFT transformation to convert time domain data into frequency domain data and obtain information about power data at different frequency components. Then, PCA is used to reduce the dimensionality of frequency domain data and extract key feature vectors. These feature vectors retain the core information of power data, reduce the amount of data, and improve data transmission and processing efficiency.

[0066] The transmission path optimization model based on reinforcement learning in the deep learning module runs on the transmission node. The agent obtains network status observations in real time through sensors, including node load, link bandwidth, delay and other information. Based on these observations, the agent selects the transmission path action through the policy network. If the selected path has high transmission efficiency and low delay, the environment gives positive rewards; otherwise, negative rewards are given. Through continuous training, the agent learns the optimal path selection strategy. When a link has an increased delay, the agent can quickly adjust the transmission path within 1 second and select a better link for data transmission, effectively reducing the transmission delay.

[0067] A cache mechanism is set up at the relay and aggregation nodes, and the cache is managed by combining the least recently used (LRU) algorithm with data timeliness. The access time and update time are recorded for each cached data block. When the cache space is insufficient, eligible data blocks are eliminated according to the cache replacement formula. The cached data is checked regularly, and for data blocks whose update time exceeds the preset threshold, if they are low-frequency access data, they are deleted; if they are high-frequency access data, the data is updated. This reduces duplicate transmission and improves system performance.

[0068] Data anomaly detection and repair implementation:

[0069] The data anomaly detection and repair mechanism based on deep learning runs on the server of the data aggregation layer. A convolutional neural network (CNN)-based anomaly detection model is built and trained using a large amount of historical normal power data and a small amount of known abnormal data. During the training process, the model parameters are continuously adjusted through the back propagation algorithm to minimize the loss function. The trained model monitors the power data in real time, compares the current data with the learned normal pattern, and once a data point deviates from the normal pattern is found, it is immediately determined as abnormal data and an alarm is issued.

[0070] For abnormal data, a generative adversarial network (GAN) is used to repair it. GAN consists of a generator and a discriminator. The generator learns the distribution of normal data and generates reasonable data related to abnormal data; the discriminator determines whether the generated data is true. Through continuous training, the generator can generate high-quality repair data and restore the integrity of the data. In the power data monitoring of a factory, the detection model promptly discovered the abnormal fluctuations of the current data, and the generative adversarial network repaired the abnormal data within 500 milliseconds to ensure data accuracy.

[0071] A real-time monitoring and dynamic feedback mechanism is built to monitor the anomaly detection and repair process in real time. The system collects the performance indicators of the model, such as accuracy and recall, every 100 milliseconds. Based on the monitoring results, the parameters of the detection and repair model are dynamically adjusted using a dynamic feedback formula. When the data characteristics change in different seasons and different power consumption periods, the mechanism can automatically adjust the model parameters to maintain high-accuracy anomaly detection and repair capabilities.

[0072] The specific implementation steps of this system are as follows:

[0073] 1. Distributed Collaborative Architecture Design

[0074] 1.1 Multi-layer distributed node architecture

[0075] Construct a multi-layer architecture including data collection layer, data relay layer and data aggregation layer. The data collection layer is composed of a large number of distributed power collection terminals (smart meters, transformer collection modules, etc.), which are divided into multiple subnets based on geographical location or power grid area. The nodes in the subnet communicate using a multi-hop wireless network protocol to form a local mesh network, which improves the reliability and coverage of data collection. The data relay layer deploys multiple high-performance relay nodes, which are responsible for fast data forwarding and routing optimization, and dynamically adjusts the data flow according to the network status. The data aggregation layer is composed of a central server cluster, which aggregates, stores, analyzes and processes data.

[0076] 1.2 Dynamic load balancing algorithm:

[0077] Node load balancing is achieved based on the improved Weighted Round Robin (WRR) algorithm. For each node n i Assign dynamic weight w i (t), comprehensively consider the CPU usage u of the node i (t), memory idle rate m i (t), network bandwidth utilization b i (t) and historical task processing success rate s iThe nodes here include relay nodes and aggregation nodes in the system. These nodes undertake different tasks in the data transmission and processing process. Relay nodes are responsible for data forwarding and routing optimization, and aggregation nodes are used to aggregate, store, analyze and process data. It is very important to reasonably allocate loads for them, which directly affects the overall performance of the system. The weight calculation formula is:

[0078]

[0079] Among them, α, β, γ, and § are weight coefficients, and α+β+γ+§ = 1. When routing data, data traffic is distributed according to node weights to ensure load balancing of each node and improve the overall performance and stability of the system.

[0080] 1.3 Data redundancy and recovery mechanism

[0081] Erasure Coding technology is used for data redundant storage and transmission. The original power collection data is divided into k data blocks D = {d1, d2, ..., d k}, generate n redundant blocks R = {r1, r2, ..., r n}, forming a (k+n) redundancy scheme. The Reed-Solomon coding algorithm is used. When recovering data, as long as any k data blocks (original blocks or redundant blocks) are obtained, the original data can be restored through the decoding algorithm, which effectively responds to node failures and data loss and ensures data integrity. With the help of distributed storage and redundant backup mechanisms, combined with dynamic load balancing algorithms, it can effectively respond to abnormal situations such as node failures and network fluctuations to ensure data integrity and availability. When some nodes or links fail, the system can automatically switch and adjust to maintain the stability of data transmission, reduce the risk of data loss, and ensure the continuous and reliable operation of the power system.

[0082] 2. Implementation of high concurrent transmission mechanism

[0083] 2.1 Multi-threaded concurrent transmission model

[0084] During data transmission, multiple threads T = {t1, t2, ..., t p} for concurrent transmission. Introducing thread priority queue and dynamic thread pool management technology, according to the data priority P d Dynamically adjust the number of threads p and thread task allocation according to the network load status. Allocate more thread resources to high-priority data. Increase the number of threads to improve the transmission rate when the network is idle. Reduce the number of low-priority threads when the network is congested to ensure critical data transmission. The thread task allocation formula is as follows:

[0085]

[0086] where t alloc (i) is thread t i The amount of tasks assigned, P d (i) is the priority of data block i, N task is the total task volume, P d (j) represents the priority of data block j. The larger the priority value, the more important or urgent the data block is.

[0087] 2.2 Adaptive Flow Control Algorithm

[0088] Design an adaptive sending window adjustment algorithm to achieve flow control, design an adaptive algorithm and intelligent strategy, and reasonably allocate network bandwidth, computing resources, and storage resources according to data priority, network status, and system resource status. While ensuring the priority transmission of key data, make full use of idle resources, improve the overall resource utilization of the system, reduce operating costs, and improve the cost performance of the system. The part responsible for data transmission in the high-concurrency transmission module is the sending end. In the transmission from the data relay layer to the data aggregation layer, the node of the data aggregation layer is the receiving end. The sending end calculates the receiving window size R according to the feedback from the receiving end. w , Round-TripTime (RTT)t RTT and network congestion level C to dynamically adjust the sending window S w With the sending rate Sr, the formula is as follows:

[0089]

[0090] Among them, α, β, and γ are adjustment coefficients, and τ is the congestion threshold. When the network congestion is light, the sending window is moderately increased; when the congestion is severe, the sending window is greatly reduced to effectively avoid data loss and congestion.

[0091] 2.3 Asynchronous event-driven transmission architecture

[0092] The transmission architecture is built based on asynchronous I / O and event-driven models. Data transmission events (sending, receiving, connection establishment / disconnection, etc.) are abstracted into independent event processing modules, which are executed asynchronously when events are triggered without blocking the main thread. When data is sent, the main thread submits the sending task and continues to process other transactions. The sending module executes the sending operation in the background according to the network status and strategy. When the operation is completed or an error occurs, the main thread is notified through the callback function, which improves the system's concurrent processing capabilities and response speed.

[0093] Therefore, the present invention uses a distributed collaborative architecture and a high-concurrency transmission mechanism to break the transmission bottleneck of the traditional architecture, greatly improve the transmission rate of power collection data, and significantly reduce transmission delays. It ensures that data can be accurately transmitted from the collection end to the processing end in a short time, meets the strict requirements of the power system for data real-time, and provides timely and reliable data support for power dispatching, real-time monitoring and other services.

[0094] 3. Intelligent Data Processing and Optimization

[0095] 3.1 Data preprocessing and feature extraction algorithm

[0096] The power data preprocessing and feature extraction at the collection terminal and transmission node adopts the method of combining Discrete Fourier Transform (DFT) and Principal Component Analysis (PCA). First, the power data sequence x(n) is transformed by DFT:

[0097]

[0098] X(k) represents the frequency domain component at frequency k after discrete Fourier transform; N represents the number of sampling points, that is, the length of the electric energy data sequence x(n); k is the frequency index, which is used to specify different frequency points in the frequency domain;

[0099] After obtaining the frequency domain information, PCA is used to reduce the dimension and extract the key feature vector F = {f1,f2,..,f m}, reduce the amount of data, retain core information, and improve transmission and processing efficiency.

[0100] 3.2 Transmission Path Optimization Model Based on Reinforcement Learning

[0101] Reinforcement learning algorithm is used to optimize the transmission path. The agent selects the transmission path action a through the policy network π(s) according to the network state observation value s (node ​​load, link bandwidth, delay, etc.), and the environment feedback reward r (path transmission efficiency improvement, delay reduction, etc.). In the intelligent transmission system of power collection data, the agent can be understood as an entity with decision-making ability. Its responsibility is to make decisions on the transmission path based on the current network state. The deep Q network (DeepQNetwork, DQN) is used to train the agent, and the loss function is:

[0102]

[0103] where y = r + γmax a′ Q(s′,a′;θ -) is the target Q value, θ is the network parameter, γ is the discount factor, s′ is the new state after executing action a, a′ is all possible actions in the new state s′, θ - is the parameter of the target network. Through continuous training, the agent learns the optimal path selection strategy and improves the transmission performance.

[0104] 3.3 Data caching and dynamic update strategy

[0105] Set up a cache mechanism at the relay and aggregation nodes, combining the least recently used (LRU) algorithm with data timeliness to manage the cache. i Record access time t access (i) and update time t update (i), the cache replacement formula is:

[0106] B replace =argmin i (t access (i))∧(t update (i) <T expire )

[0107] T expire It is a time threshold used to measure the timeliness of cached data blocks;

[0108] At the same time, cached data is checked regularly based on data update frequency and importance, and expired data is updated or deleted to reduce duplicate transmission and improve system performance.

[0109] 4. Data Anomaly Detection and Repair Based on Deep Learning

[0110] 4.1 Anomaly Detection Model Construction

[0111] An anomaly detection model based on deep learning is constructed, and a convolutional neural network (CNN) is used to model the power collection data. The model learns the normal pattern of the data and identifies the data points that deviate from the normal pattern as abnormal data. The training formula of the anomaly detection model is:

[0112]

[0113] where y i For real data, is the model's predicted data, and N is the number of data samples. By minimizing the loss function, the model can accurately identify abnormal data.

[0114] 4.2 Data repair algorithm design

[0115] Design a data repair algorithm based on deep learning, and use generative adversarial networks (GAN) or variational autoencoders (VAE) to repair abnormal data. These models generate repair values ​​similar to normal data by learning the distribution of data. The data repair formula is:

[0116]

[0117] Where D normal is normal data, D abnormal For abnormal data, The repaired data. By training the generator and the discriminator, GAN can generate high-quality repaired data and restore the integrity of the data.

[0118] 4.3 Real-time monitoring and dynamic feedback mechanism

[0119] Build a real-time monitoring and dynamic feedback mechanism to monitor the anomaly detection and repair process in real time. The system dynamically adjusts the parameters of the detection and repair model based on the monitoring results to ensure that the model can maintain high performance under different operating conditions. The dynamic feedback formula is:

[0120]

[0121] Where θ is the model parameter, η is the learning rate, is the gradient of the loss function. By dynamically updating the model parameters, the system can adapt to the dynamic changes of data and improve the accuracy and real-time performance of anomaly detection and repair.

[0122] Therefore, we integrate intelligent data processing technology and data anomaly detection and repair mechanism based on deep learning to conduct in-depth mining and analysis of power collection data. It can not only detect data anomalies in real time, but also accurately repair abnormal data, providing high-quality data support for power system operation analysis, fault diagnosis, load forecasting, etc., and helping power management to upgrade to intelligent and refined directions.

[0123] The present invention needs to further demonstrate that in the construction of a smart grid in a large city, the electric energy collection data intelligent transmission system based on distributed collaboration and high concurrency mechanism of the present invention is fully deployed. The power network of the city covers a wide range and has a variety of user types, including a large number of residential users, commercial users and industrial users, and has extremely high requirements for the collection and transmission of electric energy data.

[0124] At the data collection layer, smart meters and transformer collection modules distributed in various areas collect voltage, current, power and other energy data in real time at set time intervals (e.g., every 15 minutes). These collection terminals are interconnected through a multi-hop wireless network protocol to form a local mesh network to ensure the reliability and coverage of data collection. Even in some areas with weak signals, data can be accurately transmitted to the data relay layer through multi-hop routing.

[0125] The high-performance relay nodes of the data relay layer monitor the network status in real time. According to the dynamic load balancing algorithm, the dynamic weight is assigned to each node based on the node's CPU usage, memory idle rate, network bandwidth utilization, and historical task processing success rate. Reasonable distribution of data traffic is achieved. During peak power consumption periods, network traffic in some areas increases dramatically. The relay nodes automatically adjust data routing and disperse data traffic to nodes with lighter loads, avoiding network congestion and ensuring rapid data forwarding. At the same time, the relay nodes use an adaptive flow control algorithm to dynamically adjust the sending window and sending rate based on the receiving window size, round-trip delay, and network congestion level fed back by the receiving end. When network congestion is detected, the sending window is reduced in time, the sending rate is reduced to avoid data loss; when the network condition is good, the sending window is increased to improve transmission efficiency.

[0126] In the data aggregation layer, the central server cluster is responsible for aggregating, storing, analyzing and processing data from the relay layer. Erasure coding technology is used for data redundant storage, the original power collection data is divided into multiple data blocks, and corresponding redundant blocks are generated. In a network failure, some data blocks were lost, but the original data was successfully restored through redundant blocks and Reed-Solomon coding algorithm to ensure data integrity. In terms of data processing and optimization, the collection terminal and transmission node preprocess and extract features of power data. The method of combining discrete Fourier transform and principal component analysis is used to reduce the amount of data while retaining core information, thereby improving data transmission and processing efficiency.

[0127] The transmission path optimization model based on reinforcement learning continuously learns changes in network status and selects the optimal transmission path for data. For example, when a link has an increased delay, the agent quickly adjusts the transmission path through the policy network and selects a better link for data transmission, effectively reducing the transmission delay. The data anomaly detection and repair mechanism based on deep learning plays an important role in this system. The convolutional neural network monitors power data in real time, learns the pattern of normal data, and immediately determines it as abnormal data once a data point deviates from the normal pattern is found. In the power data monitoring of a factory, the detection model promptly discovered the abnormal fluctuation of current data, and repaired the abnormal data through the generative adversarial network to ensure the accuracy of the data. At the same time, the real-time monitoring and dynamic feedback mechanism dynamically adjusts the parameters of the detection and repair model according to the monitoring results to adapt to the dynamic changes of the data. In different seasons and different power consumption periods, the characteristics of the data will change. This mechanism can automatically adjust the model parameters to maintain high-accuracy anomaly detection and repair capabilities.

[0128] After long-term operation monitoring, the system has significantly improved the transmission efficiency and quality of power collection data. Data transmission delay has been reduced by more than 50%, data loss rate has been controlled within 0.1%, and resource utilization has been increased by 30%. Through intelligent data processing and anomaly detection and repair, it provides accurate power data support for power companies, helping them to more accurately forecast loads, diagnose faults and dispatch power, effectively improving the operation and management level of the city's smart grid, reducing operating costs, and improving power supply reliability and user satisfaction.

[0129] The above specific embodiments are only several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. An intelligent transmission system for electric energy collection data based on distributed collaboration and high concurrency mechanism, characterized in that: It includes a multi-layer distributed node architecture module, a high-concurrency transmission module, a data processing and optimization module, and a deep learning module connected in sequence, among which: The multi-layer distributed node architecture module adopts a dynamic load balancing algorithm and a data redundancy and recovery mechanism. The multi-layer distributed node architecture module includes a data acquisition layer, a data relay layer and a data aggregation layer. The data acquisition layer includes multiple distributed power acquisition terminals, which are divided into multiple subnets based on geographical location or power grid area. The nodes in the subnet communicate using a multi-hop wireless network protocol to form a local mesh network. The data relay layer deploys multiple high-performance relay nodes for forwarding data and optimizing routing, and dynamically adjusts the data flow direction according to the network status. The data aggregation layer is composed of a central server cluster to aggregate, store, analyze and process data. High-concurrency transmission module, including multi-threaded concurrent transmission model, adaptive flow control algorithm and asynchronous event-driven transmission unit, dynamically adjusts the number of threads and sending window according to network status; The data processing and optimization module pre-processes and extracts features of the electric energy data at the data collection layer and transmission nodes, and uses a reinforcement learning algorithm to optimize the transmission path, and sets a cache mechanism at the relay nodes and aggregation nodes corresponding to the data relay layer and the data aggregation layer to optimize the data; The deep learning module uses convolutional neural networks to model power collection data and uses generative adversarial networks or variational autoencoders to repair abnormal data.

2. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 1 is characterized in that: The dynamic load balancing algorithm uses an improved weighted round-robin algorithm to balance node loads. i Assign dynamic weight w i (t), considering the CPU usage u of the node i (t), memory idle rate m i (t), network bandwidth utilization b i (t) and historical task processing success rate s i的 Factors, weight calculation formula is: Among them, α, β, γ, § are weight coefficients, and α+β+γ+§=1.

3. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 1 is characterized in that: The data redundancy and recovery mechanism adopts erasure coding technology to perform data redundant storage and transmission, and divides the original power collection data into k data blocks D = {d1, d2, ..., d k }, generate n redundant blocks R = {r1, r2, ..., r n }, forming a (k+n) redundancy scheme.

4. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 3 is characterized in that: The multi-thread concurrent transmission model creates multiple threads T = {t1, t2, ..., t p }Concurrent transmission, introducing thread priority queue and dynamic thread pool management technology, according to the data priority P d Dynamically adjust the number of threads p and thread task allocation according to network load conditions; The number of priority threads is adjusted according to the network status, and the thread task allocation formula is as follows: where t alloc (i) is thread t i The amount of tasks assigned, P d (i) is the priority of data block i, N task is the total task volume, P d (j) represents the priority of data block j. The larger the priority value, the more important or urgent the data block is.

5. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 4 is characterized in that: The adaptive flow control algorithm is used to control the flow of the adaptive sending window adjustment algorithm. The part responsible for data transmission in the high-concurrency transmission module is the sending end. In the transmission from the data relay layer to the data aggregation layer, the node of the data aggregation layer is the receiving end. The sending end adjusts the receiving window size R according to the feedback of the receiving end. w , round trip delay t RTT and network congestion level C to dynamically adjust the sending window S w With the sending rate Sr, the formula is as follows: Among them, α, β, γ are adjustment coefficients, and τ is the congestion threshold.

6. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 5 is characterized in that: The asynchronous event-driven transmission unit builds a transmission architecture based on asynchronous I / O and event-driven models, abstracts data transmission events into independent event processing modules, and executes asynchronously when events are triggered without blocking the main thread.

7. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 6 is characterized in that: When the data processing and optimization module preprocesses the electric energy data and extracts features, it first performs discrete Fourier transform on the electric energy data sequence x(n): X(k) represents the frequency domain component at frequency k after discrete Fourier transform; N represents the number of sampling points, that is, the length of the electric energy data sequence x(n); k is the frequency index, which is used to specify different frequency points in the frequency domain; After obtaining the frequency domain information, the principal component analysis method is used to reduce the dimension and extract the key feature vector F = {f1,f2,..,f m }, reduce the amount of data and retain core information.

8. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 7 is characterized in that: In the reinforcement learning algorithm, the agent selects the transmission path action a through the policy network π(s) according to the network state observation value s, and the environment feedback reward r. The deep Q network is used to train the agent, and the loss function is: where y = r + γmax a′ Q(s′,a′;θ - ) is the target Q value, θ is the network parameter, γ is the discount factor, s′ is the new state after executing action a, a′ is all possible actions in the new state s′, θ - are the parameters of the target network.

9. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 1 is characterized in that: The cache mechanism combines the least recently used algorithm with data timeliness to manage the cache, and records the access time t for the cached data block Bi. access (i) and update time t update (i), the cache replacement formula is: B replace =argmin i (t access (i))∧(t update (i)<T expire ) T expire It is a time threshold used to measure the timeliness of cached data blocks; At the same time, cached data is checked regularly based on data update frequency and importance, and expired data is updated or deleted.

10. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 9 is characterized in that: The deep learning module constructs an anomaly detection model based on deep learning. The anomaly detection model based on deep learning identifies data points that deviate from the normal pattern as abnormal data by learning the normal pattern of the data. The training formula of the anomaly detection model is: where y i For real data, is the model prediction data, N is the number of data samples; When the deep learning module repairs abnormal data, it generates a repair value similar to normal data by learning the distribution of the data. The data repair formula is: Where D normal is normal data, D abnormal For abnormal data, The repaired data.

11. The electric energy acquisition data intelligent transmission system based on distributed collaboration and high concurrency mechanism according to claim 10 is characterized in that: The deep learning module also includes building a real-time monitoring and dynamic feedback mechanism to monitor the anomaly detection and repair process in real time, and dynamically adjust the parameters of the detection and repair model according to the monitoring results to ensure that the model maintains high performance under different operating conditions. The feedback formula for dynamic adjustment is: Where θ is the model parameter, η is the learning rate, is the gradient of the loss function.

Images