A microgrid intelligent and safe secondary voltage control method based on reinforcement learning

Abstract

The present invention relates to a microgrid intelligent security secondary voltage control method based on reinforcement learning, which belongs to the field of power grid technology. The method first establishes a microgrid secondary control model under hybrid network attacks. A DoS attack defense network based on deep learning is designed, an adaptive FDI attack compensation state observer is designed, compensation for unknown FDI attacks is established, and an adaptive law of the compensation signal is designed to achieve estimation of the system state under network attacks. An evaluation-execution structure is constructed based on reinforcement learning, and a virtual optimal controller and an actual optimal controller are designed in combination with the backstepping method. A neural network is used to approximate the unknown parts in the system model, evaluation network, and execution network, and a weight adjustment law is given. A log-type barrier function is introduced into the performance indicator function to constrain the output within a fixed constant interval to meet the constraint conditions. The present invention can cope with DoS attacks and FDI attacks and ensure stable control and safe operation of the microgrid secondary voltage.

Description

Technical Field

[0001] The present invention belongs to the technical field of power grids and relates to a microgrid intelligent safety secondary voltage control method based on reinforcement learning. Background Art

[0002] Microgrids organically combine multiple distributed generation units and energy storage systems within a region, fully utilizing local distributed energy resources. They offer the advantages of being economical, environmentally friendly, flexible, and reliable. Based on a power electronics inverter and rectifier connection, each distributed generation unit in a microgrid is an autonomous, intelligent entity, thus providing the foundation for adapting to more advanced intelligent control algorithms. However, microgrids lack the inertia of traditional grid generators, suffering from low inertia, volatile network structures, and parameter uncertainty. Furthermore, the output of most distributed generation units fluctuates and is intermittent due to environmental factors, posing significant challenges to maintaining the stability of the microgrid system.

[0003] The stable operation of the power system is essential for high-quality power supply. The power system is a complex, interconnected cyber-physical system. Even a small disturbance at a single node can cause multiple nodes to fail if not addressed promptly. Cascading failures caused by a single node failure are the primary cause of entire power grid collapse. In a power system, the stability of the power angle, frequency, and voltage is fundamental to its stable operation.

[0004] A microgrid is a small power system consisting of distributed generation units, energy storage devices, and loads. Under normal circumstances, a microgrid operates in parallel with the conventional distribution network. However, if a grid fault is detected or power quality fails to meet requirements, the microgrid disconnects from the grid and operates in island mode. In island mode, the primary control layer, consisting of two cascaded internal proportional-integral controllers and an external droop controller, ensures frequency and voltage stability. However, frequency and voltage inevitably deviate from their rated values, so a secondary controller is designed to compensate for these deviations.

[0005] Secondary control of microgrids is usually implemented by centralized or distributed structures. The centralized structure requires a central computing and communication unit, which is expensive and sensitive to single-point failures. In a distributed structure, each distributed generation unit only needs to communicate with adjacent generation units. In existing research results, secondary control is regarded as a consistency tracking problem and distributed secondary controllers are designed, such as distributed average proportional integral secondary controllers, which can eliminate frequency deviations and achieve desired active power sharing control. In isolated microgrids, most loads, such as motors, need to operate at rated frequency and voltage. Distributed generation units have low inertia and are easily affected by uncertainties and intermittent environmental changes. Existing methods design finite-time secondary controllers based on sliding mode control, which can accelerate convergence and achieve frequency and voltage recovery.

[0006] Due to the discontinuous nature of the controller, chattering can occur, affecting control accuracy and potentially stimulating unmodeled dynamics that can impact system stability. To avoid or reduce chattering, continuous control strategies can be implemented based on Lyapunov functions. Distributed secondary controllers have been proposed based on precise microgrid system models, but they do not account for system uncertainties such as parameter disturbances, unmeasured variables, and external load variations. In practice, measures must be taken to prevent system uncertainties from decreasing or even destabilizing the microgrid. In existing literature, supertorque and full-order distributed secondary control methods have been proposed to enhance system robustness, but both require prior knowledge of the upper bounds of uncertainty derivatives and are therefore not applicable in practical microgrid systems.

[0007] The core idea of ​​reinforcement learning is to allow an intelligent agent to interact with its environment, learning and improving its behavioral strategies through feedback from the environment in response to different behaviors. As a machine learning method, reinforcement learning can effectively solve optimal control problems for complex systems without requiring knowledge of the precise system dynamics. Furthermore, reinforcement learning considers the cost optimization during the long-term interaction between the intelligent agent or actuator and the environment, seeking the optimal control strategy that optimizes the performance function, thus conserving resources. Recently, reinforcement learning techniques have been applied to solve optimal control problems with unknown system dynamics in various control domains, such as designing robot motion control, threshold attack detection schemes, and energy-saving control. Current mainstream reinforcement learning research is typically implemented within an actor-critic framework, where an evaluator evaluates the performance of the current policy based on measured data, while the executor uses the cost function obtained by the evaluator's evaluation to improve its policy. Furthermore, by combining techniques such as neural networks, fuzzy logic control, and backstepping, the critic-executor control framework can be applied to more general systems. However, most research efforts fail to consider the reliability and security of information communication, which can significantly impact the performance of networked control systems.

[0008] The security risks posed by network attacks cannot be ignored. Without security protection, information can be easily exploited by malicious attackers during transmission, potentially causing adverse effects on the system. Therefore, establishing secure mechanisms to defend against network attacks is crucial. Typical network attacks include denial of service (DoS) and spoofing attacks. Denial of service (DoS) attacks disrupt communication between devices by disrupting network channels or attacking routing protocols. Spoofing attacks damage data by modifying the information in transmitted packets within network channels. These attacks primarily include false data injection (FDI) and replay attacks. FDI attacks inject false information to undermine the authenticity of data.

[0009] Currently, the main control strategies for dealing with DoS attacks include Robust resilient control strategies with finite update cycles, control strategies based on switching type estimators, event-triggered control strategies based on reliable attack detection mechanisms, and control strategies designed with state feedback gains. Periodic DoS attacks are typically modeled as intervals. The absence of a DoS attack within the interval indicates normal communication, while the presence of a DoS attack within the interval indicates communication interruption. In reality, the frequency of attacks is complex and variable, necessitating the study of system security control under random DoS attacks. A random DoS attack involves randomly disrupting the communication channel at any given moment. To facilitate analysis of the impact of random attacks on the communication channel, it is often assumed that the attack obeys specific statistical laws. Random DoS attacks are typically modeled as following a Bernoulli probability distribution, which is then used for controller design. Research has investigated random DoS jamming attacks by designing a link-based estimator to estimate the relative states of agents between intermittent communications, and developing an event-triggered control scheme based on relative inter-agent states.

[0010] Regarding security control strategies for FDI attacks, an adaptive control algorithm for strictly feedback nonlinear systems is proposed. For cyber-physical systems experiencing spoofing attacks and containing unknown nonlinearities, an adaptive security control strategy based on a novel coordinate transformation is proposed. A previous study investigated the resilient control of load frequency control systems under FDI attacks. This paper combined a model-based observer with a learning-based observer to jointly detect and compensate for FDI attacks in real time, eliminating the impact of abnormalities on the reconfiguration of control variables in the load frequency control system. Summary of the Invention

[0011] In view of this, the object of the present invention is to provide a microgrid intelligent safety secondary voltage control method based on reinforcement learning.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] A microgrid intelligent safety secondary voltage control method based on reinforcement learning, the method comprising the following steps:

[0014] Step 1: Establish a microgrid secondary control model under hybrid network attacks, including:

[0015] Establish a communication network model, in which the communication relationship between distributed generation units is abstracted as an undirected graph;

[0016] Model DoS attacks and FDI attacks, and establish a packet loss probability distribution model and an FDI attack signal model respectively;

[0017] The state equation of the microgrid secondary voltage control system with output constraints under hybrid network attacks is established, taking into account uncertainties such as system parameter disturbances, unknown parameters, and unmeasurable variables.

[0018] This is the foundation of the entire control method. It is necessary to first identify the security threats faced by the microgrid's secondary control system, namely DoS attacks and FDI attacks, and establish corresponding mathematical models to describe the impact of these attacks on the system. This step provides the basis for subsequent system modeling and analysis.

[0019] Step 2: Design a deep learning-based DoS attack defense network, including a convolutional neural network (CNN), a long short-term memory (LSTM) network, and an autoregressive (AR) module, to predict and compensate for system measurement value loss caused by DoS attacks.

[0020] DoS attacks can cause system measurement loss, necessitating the design of appropriate defense mechanisms to compensate for lost data. A deep learning-based DoS attack defense network, using CNN, LSTM, and AR modules, can effectively predict and compensate for lost measurements, providing accurate data for subsequent state estimation and control.

[0021] Step 3: Design an adaptive FDI attack compensation state observer. Use a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks. Design an adaptive law for the compensation signal to estimate the system state under network attacks.

[0022] FDI attacks can tamper with system measurements, leading to distorted state estimates. Therefore, an adaptive FDI attack-compensating state observer is needed to eliminate the effects of the attack and accurately estimate the system state. This observer uses a compensation mechanism based on a projection operator, which effectively compensates for the errors caused by unknown FDI attacks.

[0023] Step 4: Based on the DoS and FDI attack defense mechanism, a security optimization controller is designed. An evaluation-execution structure is constructed based on reinforcement learning. The virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The neural network is used to approximate the unknown parts of the system model, evaluation network, and execution network, and a weight adjustment law is given. Considering that the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function. The properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0024] Furthermore, the DoS attack defense network based on deep learning includes:

[0025] Convolutional neural network (CNN) module, used to extract effective features of measurement data;

[0026] Long short-term memory network (LSTM) module, used to predict its long-term development trend;

[0027] Autoregressive AR module, used to improve the adaptability of deep learning model output data to changes in input data;

[0028] The CNN module first performs feature extraction on the input measurement data to extract the features in the data;

[0029] The LSTM module receives the features extracted by the CNN module and combines its own memory capacity to predict the long-term and short-term trends of the data;

[0030] The AR module receives the output of the CNN and LSTM modules and further performs autoregressive prediction on the data to improve the model's sensitivity to the scale of the input data and adapt to the non-periodic changes in sensor measurement data under network attacks.

[0031] Furthermore, the CNN module includes a squeeze excitation block SE-CNN for increasing the weight of feature channels useful for the current task, so that the model can adaptively adjust the feature response of the channel.

[0032] Furthermore, the adaptive FDI attack compensation state observer adopts a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks, and designs an adaptive law of the compensation signal to achieve estimation of the system state under network attack.

[0033] Furthermore, the safety optimization controller constructs an evaluation-execution structure based on reinforcement learning, combines the backstepping method to design a virtual optimal controller and an actual optimal controller, uses a neural network to approximate the unknown parts of the system model, evaluation network and execution network, and gives a weight adjustment law. Considering that the system output has a symmetric time-invariant constraint, a log-type barrier function is introduced in the performance index function, and the properties of the logarithmic function are used to constrain the output within a fixed constant interval to meet the constraint conditions.

[0034] The beneficial effects of the present invention are:

[0035] (1) To address the challenges posed by the openness of information transmission in communication networks to system information security, this patent considers a microgrid secondary voltage control system subject to hybrid network attacks. DoS attacks and FDI attacks occur in the feedback channel between the output sensor and the controller. Under DoS attacks, sensor detection information is lost, undermining the availability security requirements of the smart grid. Under FDI attacks, false data information is injected into the feedback channel, undermining the integrity security requirements of the smart grid. These two network attacks are modeled, and their impact on the output variables of the controlled system is considered to derive a microgrid secondary voltage control model subject to hybrid network attacks.

[0036] (2) A prediction and compensation model based on convolutional neural networks and long short-term memory neural networks is proposed to solve the problem of system measurement value loss caused by DoS attacks. The model can extract the effective features of the measurement data and predict its long-term development trend. The squeeze excitation block is embedded in the CNN, which increases the weight of the feature channels that are useful for the current task, enabling the model to adaptively adjust the characteristic response of the channel. Compensation for missing measurements can be regarded as a time series prediction problem. In combination with the autoregressive module, it can improve the adaptability of the output data of the deep learning model to changes in the input data, thereby improving the accuracy and robustness of the predicted output.

[0037] (3) Design a security observer with adaptive compensation for FDI attacks to achieve system state observation under hybrid network attacks. FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the power system measurement information, thereby affecting the effectiveness of the power control system and threatening the safe and stable operation of the power grid. They are highly concealed and destructive. The FDI attack is parameterized and a compensation mechanism based on the projection operator is adopted to establish compensation for unknown FDI attacks. The adaptive law of the compensation signal is designed to achieve estimation of the system state under network attacks.

[0038] (4) Based on reinforcement learning, an evaluation-execution structure is constructed, and the virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The universal approximation characteristics of neural networks are used to approximate the unknown parts of the system model, evaluation network, and execution network, and a weight adjustment law is given. Considering the case where the system output has symmetric time-invariant constraints, a log-type barrier function is introduced into the performance indicator function. The properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0039] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0041] Figure 1 The structural block diagram of the secondary voltage control system under hybrid network attack;

[0042] Figure 2 This is the block diagram of the microgrid secondary voltage control scheme;

[0043] Figure 3 Defending deep learning networks against DoS attacks. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0046] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] In island mode, a microgrid's primary control layer consists of two cascaded internal proportional-integral controllers and an external droop controller to ensure frequency and voltage stability. However, frequency and voltage inevitably deviate from their rated values, necessitating the design of a secondary controller to compensate for these deviations.

[0048] To avoid or reduce chattering, continuous control strategies can be implemented based on Lyapunov functions. Distributed secondary controllers proposed based on precise microgrid system models fail to account for system uncertainties caused by system parameter disturbances, unmeasured variables, and external load variations. In practice, measures are needed to prevent system uncertainties from degrading or even destabilizing the microgrid. Reinforcement learning, as a machine learning method, can effectively address optimal control methods for complex systems without requiring precise knowledge of the system dynamics.

[0049] This patent designs a smart and secure microgrid secondary voltage control scheme. Considering a microgrid secondary voltage control system subject to hybrid network attacks, a corresponding deep learning defense network and attack compensation state observer are designed to meet the security availability and integrity requirements during information transmission. Considering the security constraints on system outputs and the model uncertainty caused by system parameter perturbations and external loads, an intelligent safety controller is designed based on a neural network and barrier function combined with an evaluation-execution reinforcement learning structure.

[0050] To address the challenges to system information security posed by the openness of communication network information transmission, this patent considers a microgrid secondary voltage control system subject to hybrid network attacks. DoS and FDI attacks occur in the feedback channel between the output sensor and the controller. Under DoS attacks, sensor detection information is lost, undermining the availability security requirements of the smart grid. Under FDI attacks, false data is injected into the feedback channel, undermining the integrity security requirements of the smart grid. These two network attacks are modeled, considering their impact on the output variables of the controlled system, and a microgrid secondary voltage control model is derived for the hybrid network attack.

[0051] A prediction and compensation model based on convolutional neural networks and long short-term memory neural networks is proposed to address the problem of measurement loss caused by DoS attacks. This model extracts effective features from measurement data and predicts its long-term trends. By embedding a squeeze-excitation block into the CNN, it increases the weights of feature channels that are useful for the task at hand, enabling the model to adaptively adjust the channel's characteristic response. Compensating for missing measurements can be viewed as a time series prediction problem. Incorporating an autoregressive module improves the adaptability of the deep learning model's output data to changes in the input data, thereby enhancing the accuracy and robustness of the predicted output.

[0052] A security observer with adaptive compensation for FDI attacks is designed to enable system state observation under hybrid network attacks. FDI attacks, by tampering with power system measurement information, can successfully bypass the grid's bad data detection mechanisms, thereby affecting the effectiveness of power control systems and threatening the safe and stable operation of the grid. These attacks are highly concealed and destructive. By parameterizing FDI attacks and adopting a compensation mechanism based on projection operators, we establish compensation for unknown FDI attacks and design an adaptive law for the compensation signal to estimate the system state under network attacks.

[0053] Based on reinforcement learning, an evaluation-execution architecture is constructed, and backstepping is used to design both virtual and actual optimal controllers. The universal approximation properties of neural networks are exploited to approximate the unknown components of the system model, evaluation network, and execution network, and a weight adjustment law is proposed. Considering the case where the system output is subject to symmetric time-invariant constraints, a log-type barrier function is introduced into the performance indicator function. Leveraging the properties of logarithmic functions, the output is constrained to a fixed constant interval to satisfy the constraints.

[0054] This patent designs a smart and secure microgrid secondary voltage control scheme. Considering a microgrid secondary voltage control system subject to hybrid network attacks, a corresponding deep learning defense network and attack compensation state observer are designed to meet the security and integrity requirements of information transmission. Considering the security constraints on system outputs and the model uncertainty caused by system parameter perturbations and external loads, an intelligent safety controller is designed based on a neural network and barrier function combined with an evaluation-execution reinforcement learning structure.

[0055] To address the challenges to system information security posed by the openness of communication network information transmission, this patent considers a microgrid secondary voltage control system subject to hybrid network attacks. DoS and FDI attacks occur in the feedback channel between the output sensor and the controller. Under DoS attacks, sensor detection information is lost, undermining the availability security requirements of the smart grid. Under FDI attacks, false data is injected into the feedback channel, undermining the integrity security requirements of the smart grid. These two network attacks are modeled, considering their impact on the output variables of the controlled system, and a microgrid secondary voltage control model is derived for the hybrid network attack.

[0056] A prediction and compensation model based on convolutional neural networks and long short-term memory neural networks is proposed to address the problem of measurement loss caused by DoS attacks. This model extracts effective features from measurement data and predicts its long-term trends. By embedding a squeeze excitation block into the CNN, it increases the weights of feature channels that are useful for the task at hand, enabling the model to adaptively adjust the channel's characteristic response. Compensating for missing measurements can be viewed as a time series prediction problem. Incorporating an autoregressive module improves the adaptability of the deep learning model's output to changes in input data, enhancing the accuracy and robustness of the predicted output.

[0057] A security observer with adaptive compensation for FDI attacks is designed to enable system state observation under hybrid network attacks. FDI attacks, by tampering with power system measurement information, can successfully bypass the grid's bad data detection mechanisms, thereby affecting the effectiveness of power control systems and threatening the safe and stable operation of the grid. These attacks are highly concealed and destructive. By parameterizing FDI attacks and adopting a compensation mechanism based on projection operators, we establish compensation for unknown FDI attacks and design an adaptive law for the compensation signal to estimate the system state under network attacks.

[0058] Based on reinforcement learning, an evaluation-execution architecture is constructed, and backstepping is used to design both virtual and actual optimal controllers. The universal approximation properties of neural networks are exploited to approximate the unknown components of the system model, evaluation network, and execution network, and a weight adjustment law is proposed. Considering the case where the system output is subject to symmetric time-invariant constraints, a log-type barrier function is introduced into the performance indicator function. Leveraging the properties of logarithmic functions, the output is constrained to a fixed constant interval to satisfy the constraints.

[0059] 1. Establishment of microgrid secondary control model under hybrid network attacks

[0060] 1.1 Communication Network

[0061] Consider N distributed generation units, and the communication between them is described by an undirected graph G = (v, ε). All distributed generation units correspond to the node set v = {1, 2, ..., N}. The communication relationship between distributed generation units is abstracted as the edges in the graph, forming the set of edges (v i ,v j ) represents the edge between nodes i and j. If v i You can v j Send information, then (v i ,v j )∈ε. A=[a ij ] N×N It is called the adjacency matrix of the graph. When (v i ,v j )∈εwhen a ij =a ji = 1. Otherwise a ij =a ji =0, and a ii = 0. Define the in-degree matrix D = diag{d1, d2, ..., d N},middle The communication network between distributed generation units is undirected and connected, and there is a path between any two nodes. Assume that the virtual leader is node υ0, and define the leader adjacency matrix B = diag{b1,b2,...,b N}, if node v0 and node v i If there is an edge between them, then b i =1, otherwise b i =0.

[0062] 1.2 Description of Hybrid Attacks

[0063] While communication networks offer immense convenience, the open nature of information transmission also poses unprecedented challenges to system security. Typical network attacks include denial of service (DoS) and spoofing attacks. DoS attacks disrupt communication between devices by disrupting network channels or attacking routing protocols. Spoofing attacks damage data by modifying the information in transmitted packets within network channels. These attacks primarily include false data injection (FDI) and replay attacks. FDI attacks inject false information to undermine data authenticity.

[0064] Consider DoS and FDI attacks that occur in the feedback channel between the output sensor and the controller, while the remaining control system channels are secure and reliable. DoS attacks deplete network resources and disrupt network communications, preventing sensor measurement information from being properly transmitted to the control center and potentially paralyzing the power grid. FDI attacks are subtle and designed to tamper with transmitted information, reducing the economic efficiency of the power grid.

[0065] When a DoS attack occurs, input signals in the communication network are blocked and lost, affecting the observation efficiency of the safety observer and the control performance of the controller. DoS attacks attempt to block transmission channels, preventing data from reaching the controller at a specific time. Blocking the interconnection of communication modules will cause data transmission failures and even destabilize the system.

[0066] Define a time-varying parameter λ i ∈{0,1} to indicate whether the data packet is lost, then the attacked control output signal can be expressed as According to the packet loss situation, a random process is used to model it. The probability distribution is obtained as follows:

[0067]

[0068] Where λ∈[0,1] represents the packet arrival rate, so we can get λ i The expectation is Ε(λ i )=λ.

[0069] During data transmission, the arrival rate of error-free data packets depends on the signal-to-noise ratio (SNR) of the transmitted signal at the receiving end. When a malicious DoS attack occurs, the SNR of the communication channel can be expressed as:

[0070]

[0071] Where m1∈(0,1] is the channel gain between the transmitter and the receiver, m2∈(0,1] is the attacker's interference gain, P is the transmitter's transmission power, J is the attacker's interference power, and N is the background noise. Based on the signal-to-noise ratio of the communication channel under DoS attack, the packet arrival rate can be expressed as:

[0072]

[0073] in c>0 is a constant.

[0074] Considering that FDI attacks can cause system performance degradation or even instability by destroying data accuracy, this patent is dedicated to designing secure observers and controllers to ensure the stability of networked control systems even when FDI attacks occur. When an FDI attack occurs, false information will be injected into the system's output signal, causing data accuracy to be destroyed. The attacked output is represented by in is the output variable under FDI attack, a i It is a signal of FDI attack.

[0075] 1.3 System model establishment

[0076] Considering the scenario where a microgrid's secondary control system is simultaneously subject to a hybrid DoS and FDI cyberattack, and considering the security of the system control process, the system output is constrained. Microgrids lack the inertia of traditional grid generators and have disadvantages such as low inertia, volatile network structure, and parameter uncertainty. Furthermore, the output of most distributed generation units is fluctuating and intermittent due to environmental factors, posing a significant challenge to maintaining microgrid system stability. In summary, considering the uncertainty of the controlled object, such as parameter perturbations, unknown parameters, and unmeasurable variables, a microgrid secondary voltage control system with output constraints is modeled under hybrid cyberattacks.

[0077] The block diagram of a microgrid control system designed to withstand a hybrid DoS and FDI cyberattack is shown in the figure. The primary control layer consists of two cascaded internal proportional-integral controllers and an external droop controller to ensure frequency and voltage stability. However, frequency and voltage inevitably deviate from their rated values, so a secondary controller is designed to compensate for these deviations. Droop control, by combining the output of different distributed generators, enables power self-consistency among them. As an effective distributed control method that does not rely on a communication network, it is widely used in microgrids.

[0078] Microgrid models such as Figure 1 As shown, each distributed generation unit consists of a three-phase inverter with a DC power supply, an inductor-capacitor filter L fi and C fi , output connector R ci and L ci , and the interconnection link impedance R li and L liThe controller is designed on the two-phase rotating dq coordinate system. The controller consists of a two-layer structure, including a primary controller and a secondary controller. This is the system output when it is attacked by a mixed DoS and FDI network attack.

[0079] The function of the primary controller is to keep the output voltage and frequency stable within a preset range. However, as the load conditions of the microgrid change, the primary controller cannot prevent the frequency and voltage from deviating from the rated values. Therefore, a secondary controller is required to compensate for the deviation of voltage and frequency.

[0080] The primary controller includes external droop control and internal voltage and current control. The droop control indicates the reference frequency and reference voltage of the i-th distributed generation unit. The voltage and current controllers enable the three-phase inverter to track the reference frequency and voltage values ​​provided by the droop controller.

[0081] The dynamic equations of the power control module using droop control can be expressed as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Among them, P i and Q i Indicates the cutoff frequency is ω ci Instantaneous active power and reactive power of the low-pass filter, and is the droop coefficient of frequency and voltage in primary droop control, v ni and ω ni are the expected voltage and frequency of the i-th generating unit, and for The dq component, v odi and v oqi andi odi and i oqi are the output voltage and output current of the i-th power generation unit in the dq reference coordinate system. The time-varying parameter λ i ∈{0,1} to indicate whether the data packet is lost when attacked by DoS, a i (t) is the FDI attack signal, i.e., the false information injected into the output signal.

[0088] The control of voltage and current is usually achieved using the PI control method, and the differential equations of the voltage and current controllers are given as follows:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] where i ldi ,i lqi ,v ldi ,v lqi i li and v li The d-axis and q-axis components, K PVi ,K IVi ,K PCi ,K ICi are the proportional and integral gains for voltage and current control, ω b is the nominal angular frequency, F i is a constant gain value.

[0096] The dynamic equations of the output LC filter and output connector are described by the following differential equations.

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] where i ldi and i lqi and v bdi and v bqi They are the inverter output current i li and bus voltage v bi The dq components of

[0104] The output right Differentiate to obtain the output voltage of the voltage source inverter With the droop controller input voltage v ni The relationship between them is as follows:

[0105]

[0106] Simplifying the large signal model into a second-order system, the system state equation for secondary voltage control under hybrid network attacks is obtained as follows:

[0107]

[0108]

[0109]

[0110] The control input signal is v ni , is the output of the DoS and FDI hybrid network attack, λ i ∈{0,1} represents the loss of data packets under DoS attack, a i It is a signal of FDI attack. is the output variable of the controlled system under hybrid attack, function f i and g i as follows:

[0111]

[0112]

[0113] Parameter disturbances, unknown parameters and unmeasured variables of the controlled system will cause f i Uncertain coefficients cause deviations in the system dynamic model and reduce control performance. Circuit parameter changes, such as the inductor capacitor filter L fi and C fi In practical applications, it is difficult to obtain the output connector parameter R ci 、L ci , and the voltage variable v bdi , unknown parameters and unmeasured variables appear. In particular, the inductance and capacitance parameters are of small magnitude, resulting in f i The coefficient disturbance deviation is large. Due to the existence of uncertainty factors and interference, such as the voltage fluctuation on the public line caused by the change of local load in the system, ξ i It represents the uncertainty of the system model caused by parameter disturbance.

[0114] 2. Design of Intelligent Safety Secondary Voltage Controller

[0115] In summary, considering that the controlled object is subject to a hybrid network attack of DoS and FDI, and considering that the system has output constraints, and that the controlled object may have uncertainties such as parameter disturbances, unknown parameters and unmeasurable variables, an intelligent and safe controller is designed for the microgrid secondary control system. The design scheme is as follows: Figure 2 shown.

[0116] 2.1 DoS Attack Defense Network Based on Deep Learning

[0117] DoS attacks transmit destructive information to the power system's communication network, depleting network resources and disrupting communications, preventing the control center from receiving system measurement data. DoS attackers launch malicious attacks, preventing the control center from timely and reliable access to and use of system measurement data. In real life, the catastrophic consequences can be cascading power outages, impacting grid stability and causing economic losses.

[0118] DoS attacks disrupt the communication channels of the controlled systems, resulting in the loss of system output sensor measurement data transmitted to the secondary control center. For power grids that rely on measurement data for state estimation, the loss of measurement data will reduce the accuracy of the estimation. Considering the dynamic nonlinear characteristics of power systems and the effectiveness of deep neural networks in fitting nonlinear functions, deep learning is used to predict and compensate for the lost measurement values.

[0119] A hybrid neural network model based on deep learning, consisting of CNN, LSTM and AR modules, is used to predict and compensate for the measurement loss of power system status under DoS attacks. The network structure Figure 3 shown.

[0120] SE-CNN incorporates a squeeze-excitation block into the CNN to increase the weight of feature channels useful for the current task. The CNN and LSTM modules are used to extract short-term local features between measurements and predict long-term trends. The AR model is used to increase the neural network model's sensitivity to the scale of the input data, addressing the acyclical nature of sensor measurements under cyber attacks and improving prediction performance. Each module of the hybrid neural network simultaneously receives the same system measurement as input and calculates the result. The resulting features are concatenated using the Concat function, and then a fully connected layer is used to obtain the predicted system measurement value. Finally, the predicted value replaces the missing data in the actual measurement value, thus achieving compensation.

[0121] Assume that the actual measurement value is Y. When the system suffers a DoS attack, the system measurement value will be randomly lost during the transmission process, and the measurement value collected by the control center is Y. r , need to compensate Y r To ensure the accuracy of the state estimation results, the missing measurement values ​​in the compensation matrix are defined as The goal of measurement value compensation is expressed as follows, where Θ is the set of model parameters:

[0122]

[0123] The calculation of each time step of LSTM is as follows:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] Where σ is the sigmoid function, tanh is the hyperbolic tangent function, the symbol * is the Hadamard multiplication, W is the weight matrix, b is the bias, g t is the activation vector. The LSTM model is composed of the input word v t , cell state Temporary cell state Hidden layer state Forget Gate Memory Gate and output gate The calculation process is to forget the old information in the cell state and remember the new information, so that the information useful for subsequent calculations can be transmitted. And the hidden state is output at each time step. The forgetting, memory and output processes are controlled by the hidden state of the previous moment. and the current input v t The calculated forget gate, memory gate and output gate are used to control it.

[0131] A single LSTM network cannot fully meet the forecasting needs of multivariate, complex time series scenarios. Therefore, the SE-CNN module is added to extract features from multiple complex time series and improve forecasting performance. The module consists of three stacked convolutional blocks with skip-connected structures. The first and second convolutional blocks include a convolutional layer, a batch normalization layer, and an SE layer, while the third convolutional block includes a convolutional layer and a batch normalization layer. The SE block is equivalent to adding an attention mechanism to the channel dimension and is crucial for improving the model's performance on complex datasets. Using a fully connected network that automatically learns based on loss, a new neural network is used to determine the importance of each channel in the feature map. Each feature is then assigned a weight, which modulates the importance of each feature channel, with the neural network prioritizing channels with high weights.

[0132] F tr As a convolution operator, its input is X∈R W′×H′×C′ , the output is U∈R W×H×C , expressed as U=[u1,u2,...,u cf ], the convolution process can be expressed as

[0133]

[0134] Where * represents the convolution operation, Is a two-dimensional space kernel, representing v cf A single channel of acts on the corresponding channel of X. Next, the weights of channel features are adaptively adjusted through squeezing, excitation, and scaling.

[0135] The squeeze operation obtains global information through a global average filter, and the time dimension feature K of each channel time series data is pooled by the global average pooling F sq Compressed into a real number, this real number is the channel dimension statistic z, the cth element of z is z cf :

[0136]

[0137] The excitation operation implements the function of adaptive recalibration, using a Bottleneck structure consisting of two fully connected layers to simulate the correlation between channels and output the same number of weight values ​​s as the input features:

[0138] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(z,W1z)) (2.10)

[0139] Here, δ represents the ReLU function, W1 and W2 are the parameters of the dimensionality reduction and dimensionality increase layers, respectively. This part first compresses the features through a fully connected layer, then uses a ReLU layer for nonlinear operations, then uses a fully connected layer to restore the features, and finally activates with a sigmoid function.

[0140] Finally, s is multiplied by the previous convolution layer in spatial position, and the normalized weight is weighted to the features of each channel. Finally, the output of the module is:

[0141]

[0142] Among them F scale (u cf ,s cf ) represents the channel multiplication between the feature map ucf and the scale scf,

[0143] The measurement data of sensors under cyber attacks changes continuously with a non-periodic characteristic. The LSTM and SE-CNN models are insensitive to changes in the input scale. Adding an autoregressive module improves the prediction effect. The AR model formula is:

[0144]

[0145] in is the prediction result of AR module, W a is the coefficient of the AR module, q a is the size of the input window.

[0146] The LSTM module, SE-CNN module, and AR module simultaneously receive the same measurement input and calculate their respective results. The Concat function is used to concatenate the features of the results, and then the prediction results are obtained through the fully connected layer.

[0147] The actual measurement value sent at each moment is defined as Y, and the attack matrix is ​​represented by A = random(Y,δ). The measurement value received by the control center is expressed as:

[0148] Y r =A.*Y (2.13)

[0149] A contains only two values: 0 and 1. When it is 0, it means that it is under attack and the measurement data will be lost in the data received by the control center.

[0150] Y r Input the hybrid neural network model to get the predicted value of the measured value

[0151]

[0152] The missing measurements are replaced with the predicted values ​​and combined with the non-missing measurements to form the final compensation matrix Y:

[0153]

[0154]

[0155] Compensated measurement values Input to the central estimator to estimate the state of the power system.

[0156] Output voltage v odi When a DoS attack occurs during the transmission to the secondary control center through sensor detection There will be data loss, and the compensation measurement value obtained by using the DoS attack defense network is:

[0157]

[0158] Output from a hybrid network attack After using the deep learning defense network to predict and compensate for the missing data values, the output is

[0159] 2.2 Adaptive FDI attack compensation state observer

[0160] FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the power system measurement information, thereby affecting the power system state estimation results and threatening the safe and stable operation of the power grid. They are highly concealed and destructive. After the system output is attacked by a hybrid network, the output voltage After passing through the DoS attack defense network, the voltage input to the control center is

[0161] a i The FDI attack on the system can be parameterized as:

[0162]

[0163] The unknown time-varying gain w i (t)≠-1, and satisfies and Are all unknown positive real numbers, define the function:

[0164]

[0165] Among them sup t∈[0,∞) |ψ i |<∞, The system output has the following relationship:

[0166]

[0167] By adopting a compensation mechanism based on projection operators, designing an adaptive attack compensation observer and an adaptive law for the compensation signal, we can estimate the system state under network attacks. We also establish attack compensation for unknown FDI attacks, so that the compensated state converges to the real system state.

[0168] The system state observer is designed as:

[0169]

[0170] where l i,1 , l i,2 are the parameters of the state observer, It is ψ i Estimates, Estimated output value of hybrid network attack is the compensation measurement value output by the DoS attack defense network. i There is uncertainty under the influence of factors such as parameter disturbance and external load changes, and a neural network structure can be used for approximation:

[0171]

[0172] use approximate Adjustment law design The design parameter γ f,i >0,σ i >0,z i,2 is the error value between the state and the virtual controller. The specific design method is given in the next section.

[0173] The designed state observer has the ability to compensate for FDI network attacks, where the compensation signal adaptive law The design of affects the system's observation capability and tracking performance. The adaptive law is designed as follows:

[0174]

[0175] in is a compensation signal applied to FDI attacks The projection norm bound on , the projection operator m i is a positive real number. q i >0,L i =[l i,1 l i,2 ] T .

[0176] Introduce the projection operator to transform the parameter ψ i It is restricted to the set convex set to ensure uniform boundedness. Therefore, an attack compensation observer is designed based on the projection operator, which can estimate the unknown time-varying gain generated by network attacks.

[0177] 2.3 Safety optimization controller design

[0178] Combining the knowledge of reinforcement learning and neural networks, an optimized coordinated controller is designed for the microgrid secondary voltage control multi-agent system based on the optimized backstepping controller design method to achieve rated voltage tracking and minimize the performance indicator function.

[0179] Defining the Error Signal is the optimal virtual control signal.

[0180] first step:

[0181] The tracking error signal is derived as follows:

[0182]

[0183] Define consistent tracking error:

[0184]

[0185] where N i is the neighbor node of the i-th agent, and the dynamic equation of the consistency error is:

[0186]

[0187] in

[0188] Considering symmetric time-invariant output constraints k ci is a known positive real number, then the consistent tracking error has the constraint -k bi <e i <k bi ,The optimal performance index function is defined as shown below. A log barrier function is introduced into the performance index to ensure that the system tracking error does not exceed the constraint interval.

[0189]

[0190] Among them, the parameter h i >0, As an optimal virtual control Taking the time derivative of both sides of the performance index function, we get the HJB equation:

[0191]

[0192] Find the partial derivative of the HJB equation to the optimal virtual controller, that is have to:

[0193]

[0194] Will Breaks down to:

[0195]

[0196] where β i is a positive real number designed, and Will Substituting the optimal virtual control law expression, we get:

[0197]

[0198] is an unknown and continuous function, which is approximated by a neural network and expressed as follows:

[0199]

[0200] in is the ideal neural network weight, q is the number of neurons, is the basis function vector, is the approximation error. Since the weight matrix is unknown, so the optimal virtual control law is not available. Based on the reinforcement learning method, the evaluation network and the execution network are constructed to derive the available control law.

[0201] The evaluation network designed to evaluate the controller performance is expressed as follows:

[0202]

[0203] in Yes Estimates, Is the evaluation network weight, the designed update law is as follows:

[0204]

[0205] The evaluation network design parameter γ c,i1 >0.

[0206] The action network designed for execution control is as follows:

[0207]

[0208] in is the action network weight, and its update law is as follows:

[0209]

[0210] The action network design parameter γ a,i1 >0.

[0211] Will Substituting into the HJB equation, we get:

[0212]

[0213] like Maintains and has a unique solution, which is equivalent to:

[0214]

[0215] Weight value According to the update law, it is continuously adjusted until Finally, the optimal virtual control is obtained.

[0216] Step 2:

[0217] definition The error dynamic equation is derived:

[0218]

[0219] Define the optimal performance indicator function:

[0220]

[0221] in, For the optimal actual controller, the time derivative of both sides of the performance index function is obtained, and the HJB equation is obtained as follows:

[0222]

[0223] Find the partial derivative of the HJB equation to the optimal virtual controller, that is have to:

[0224]

[0225] Will Decompose into the following form:

[0226]

[0227] where γ i is a positive real number designed, and Will Substituting into the optimal control law expression, we get:

[0228]

[0229] f i and is an unknown and continuous function, which is approximated by a neural network and expressed as follows:

[0230]

[0231] in and is the ideal neural network weight, q is the number of neurons, S f,i and is the basis function vector, ε f,i and is the approximation error. Since the weight matrix and is unknown, so the optimal control law is not available. Based on the reinforcement learning method, the evaluation network and the execution network are constructed to derive the available control law.

[0232] The evaluation network for evaluating control performance and the action network for executing control are designed as follows:

[0233]

[0234] in Yes Estimates, is the identification network weight, is the evaluation network weight, is the action network weight, and the update law is designed as follows:

[0235]

[0236]

[0237]

[0238] The identification network design parameter γ f,i >0,σ i >0, evaluate network design parameter γ c,i2 >0, action network design parameter γ a,i2 >0.

[0239] The design of the microgrid voltage secondary controller under hybrid network attack is now completed. The algorithm flow is shown in the following pseudo code:

[0240]

[0241]

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A microgrid intelligent safety secondary voltage control method based on reinforcement learning, characterized by: The method comprises the following steps: Step 1: Establish a microgrid secondary control model under hybrid network attacks, including: Establish a communication network model, in which the communication relationship between distributed generation units is abstracted as an undirected graph; Model DoS attacks and FDI attacks, and establish a packet loss probability distribution model and an FDI attack signal model respectively; The state equation of the microgrid secondary voltage control system with output constraints under hybrid network attacks is established, taking into account uncertainties such as system parameter disturbances, unknown parameters, and unmeasurable variables. Step 2: Design a deep learning-based DoS attack defense network, including a convolutional neural network (CNN), a long short-term memory (LSTM) network, and an autoregressive (AR) module, to predict and compensate for system measurement value loss caused by DoS attacks. Step 3: Design an adaptive FDI attack compensation state observer. Use a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks. Design an adaptive law for the compensation signal to estimate the system state under network attacks. Step 4: Based on the DoS and FDI attack defense mechanism, a security optimization controller is designed. An evaluation-execution structure is constructed based on reinforcement learning. The virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The neural network is used to approximate the unknown parts of the system model, evaluation network, and execution network, and a weight adjustment law is given. Considering that the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function. The properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

2. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1 is characterized in that: The deep learning-based DoS attack defense network includes: Convolutional neural network (CNN) module, used to extract effective features of measurement data; Long short-term memory network (LSTM) module, used to predict its long-term development trend; Autoregressive AR module, used to improve the adaptability of deep learning model output data to changes in input data; The CNN module first performs feature extraction on the input measurement data to extract the features in the data; The LSTM module receives the features extracted by the CNN module and combines its own memory capacity to predict the long-term and short-term trends of the data; The AR module receives the output of the CNN and LSTM modules and further performs autoregressive prediction on the data to improve the model's sensitivity to the scale of the input data and adapt to the non-periodic changes in sensor measurement data under network attacks.

3. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 2 is characterized in that: The CNN module includes a squeeze-excitation block SE-CNN for increasing the weight of feature channels useful for the current task, enabling the model to adaptively adjust the feature response of the channel.

4. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1, characterized in that: The adaptive FDI attack compensation state observer adopts a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks, and designs an adaptive law of the compensation signal to achieve the estimation of the system state under network attack.

5. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1, characterized in that: The safety optimization controller constructs an evaluation-execution structure based on reinforcement learning, combines the backstepping method to design a virtual optimal controller and an actual optimal controller, uses a neural network to approximate the unknown parts of the system model, evaluation network and execution network, and provides a weight adjustment law. Considering the symmetric time-invariant constraint of the system output, a log-type barrier function is introduced into the performance index function, and the properties of the logarithmic function are used to constrain the output within a fixed constant interval to meet the constraint conditions.

Images