Microgrid intelligent safety secondary voltage control method based on reinforcement learning

Abstract

The invention relates to a micro-grid intelligent safety secondary voltage control method based on reinforcement learning, and belongs to the technical field of power grids. The method comprises the following steps: firstly, establishing a micro-grid secondary control model under hybrid network attack; a DoS attack defense network based on deep learning is designed, a self-adaptive FDI attack compensation state observer is designed, compensation for unknown FDI attacks is established, a self-adaptive law of compensation signals is designed, and estimation of the system state under network attacks is achieved. An evaluation-execution structure is constructed based on reinforcement learning, a virtual optimal controller and an actual optimal controller are designed in combination with a backstepping method, unknown parts in a system model, an evaluation network and an execution network are approximated by using a neural network, and a weight adjustment law is given. And introducing a log type barrier function into the performance index function, and constraining the output in a fixed constant interval to meet the constraint condition. The method can cope with DoS attacks and FDI attacks, and ensures stable control and safe operation of the secondary voltage of the micro-grid.

Description

Technical Field

[0001] The present invention belongs to the technical field of power grids and relates to a microgrid intelligent safety secondary voltage control method based on reinforcement learning. Background Art

[0002] Microgrids organically combine multiple distributed power generation units and energy storage systems in a region, which can make full use of local distributed energy and have the advantages of being economical, environmentally friendly, flexible and reliable. Microgrids based on the connection of power electronic devices inverters and rectifiers, in which each distributed power generation unit is an autonomously controlled intelligent entity, have the basis for adapting more advanced intelligent control algorithms. However, microgrids lack the inertia of traditional power grid generators and have disadvantages such as low inertia, variable network structure, and uncertain parameters. At the same time, the output of most distributed power generation units is affected by environmental factors and is volatile and intermittent, which poses a huge challenge to maintaining the stability of the microgrid system.

[0003] The stable operation of the power system is the guarantee of high-quality power supply. The power system is a complex cyber-physical system with internal interconnections. If a small disturbance in a single node is not handled in time, it is likely to cause multiple nodes to fail one after another. The cascading failure caused by a single node failure is the main reason for the collapse of the entire large power grid. In the power system, the power angle, frequency and voltage stability of the power grid are the basis for the stable operation of the power system.

[0004] A microgrid is a small power system composed of distributed generation units, energy storage devices, and loads. Under normal circumstances, the microgrid is connected to the conventional distribution network. When a grid fault is detected or the power quality does not meet the requirements, the microgrid will be disconnected from the grid and operate in an island mode. In the island mode, the primary control layer consists of two cascaded internal proportional-integral controllers and an external droop controller to ensure the stability of frequency and voltage. However, the frequency and voltage will inevitably deviate from the rated value, so a secondary controller is designed to compensate for the deviation.

[0005] Secondary control of microgrids is usually implemented by centralized or distributed structures. The centralized structure requires a central computing and communication unit, which is expensive and sensitive to single-point failures. In a distributed structure, each distributed generation unit only needs to communicate with adjacent generation units. In existing research results, secondary control is regarded as a consistency tracking problem and distributed secondary controllers are designed, such as distributed average proportional integral secondary controllers, which can eliminate frequency deviations and achieve desired active power sharing control. In an island microgrid, most loads, such as motors, need to operate at rated frequency and voltage. Distributed generation units have low inertia and are easily affected by uncertainty and intermittent environmental changes. Existing methods design finite-time secondary controllers based on sliding mode control, which can accelerate convergence and achieve frequency and voltage recovery.

[0006] Due to the discontinuity of the controller, there is a chattering problem, which affects the control accuracy and may also stimulate unmodeled dynamics to affect the system stability. To avoid or reduce the chattering problem, a continuous control strategy can be implemented based on the Lyapunov function. A distributed secondary controller is proposed based on an accurate microgrid system model, but the system uncertainty, such as parameter disturbances, unmeasurable variables, external load changes, etc., is not considered. In practice, measures need to be taken to prevent the system uncertainty from being reduced or even destroying the stability of the microgrid. In the existing literature, super torque and full-order distributed secondary control methods have been proposed to enhance the robustness of the system, but both require prior knowledge of the upper bound of the uncertainty derivative and cannot be applied in actual microgrid systems.

[0007] The core idea of ​​reinforcement learning is to let the agent interact with the environment, and learn and improve its own behavior strategy through the feedback signals of the environment to different behaviors. As a machine learning method, reinforcement learning can well solve the optimization control method of complex systems without knowing the precise system dynamics. And reinforcement learning considers the cost optimization of optimizing the agent or actuator in the process of long-term interaction with the environment, and seeks the optimal control strategy that can optimize the performance function, which has the characteristics of saving resources. Recently, reinforcement learning technology has been used to solve the optimal control of unknown system dynamics in different control fields, such as designing robot motion control, threshold attack detection scheme, energy-saving control, etc. The mainstream research on reinforcement learning today is usually implemented on an Actor-Critic structure, where the evaluator evaluates the performance of the current strategy based on the measured data, and the executor uses the cost function value obtained by the evaluator to improve the strategy. Furthermore, by combining technologies such as neural networks, fuzzy logic control and backstepping, the evaluation-executive control method framework can be applied to more general systems. However, in most research results, the reliability and security of information communication are not considered, which will have a significant impact on the performance of network control systems.

[0008] The security issues caused by network attacks cannot be ignored. Without security protection, information can be easily exploited by malicious attackers during transmission and cause adverse effects on the system. It is particularly important to establish a secure mechanism to defend against network attacks. Typical network attacks include denial of service attacks and spoofing attacks. Denial of service attacks (DoS) block communications between devices by interfering with network channels or attacking routing protocols. Spoofing attacks damage data by modifying the information of transmission packets in network channels. They mainly include false data injection (FDI) and replay attacks. FDI attacks damage the authenticity of data by injecting false information.

[0009] The main control strategies for dealing with DoS attacks nowadays include Robust elastic control strategy with limited update cycle, control strategy based on switching type estimator, event-triggered control strategy based on reliable attack detection mechanism and control strategy of designing state feedback gain. Periodic DoS attacks are usually modeled in the form of intervals. When there is no DoS attack in the interval, it means that normal communication can be achieved. When there is a DoS attack in the interval, it means that communication will be interrupted. In actual situations, the frequency of attacks is complex and changeable. It is necessary to study the security control problem of the system under random DoS attacks. Random DoS attacks refer to attacks that randomly launch damage to the communication channel at any time. In order to facilitate the analysis of the impact of random attacks on the communication channel, it is usually assumed that the attack obeys specific statistical laws. Random DoS attacks are usually modeled as obeying the Bernoulli probability distribution form, and then the controller is designed. Some literature studies the random DoS interference attack, estimates the relative agent state between intermittent communications by designing a link-based estimator, and develops an event-triggered control scheme based on the relative agent state.

[0010] Regarding the safety control strategy for FDI attacks, an adaptive control algorithm for handling FDI attacks is proposed for strictly feedback nonlinear systems. For cyber-physical systems that encounter deception attacks and contain unknown nonlinearities, an adaptive safety control strategy based on a new coordinate transformation is proposed. There is a literature that studies the elastic control problem of load frequency control systems under FDI attacks. This paper uses a combination of model-based observers and learning-based observers to jointly detect and compensate for FDI attacks in real time, eliminating the impact of reconfiguration of control quantities when abnormalities occur in load frequency control systems. Summary of the invention

[0011] In view of this, an object of the present invention is to provide a microgrid intelligent safety secondary voltage control method based on reinforcement learning.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] A microgrid intelligent safety secondary voltage control method based on reinforcement learning, the method comprising the following steps:

[0014] Step 1: Establish a microgrid secondary control model under hybrid network attacks, including:

[0015] Establish a communication network model, in which the communication relationship between distributed generation units is abstracted as an undirected graph;

[0016] Model DoS attacks and FDI attacks, and establish a packet loss probability distribution model and an FDI attack signal model respectively;

[0017] The state equation of the microgrid secondary voltage control system with output constraints under hybrid network attacks is established, taking into account uncertainty factors such as system parameter disturbances, unknown parameters and unmeasurable variables;

[0018] This is the basis of the entire control method. It is necessary to first clarify the security threats faced by the microgrid secondary control system, namely DoS attacks and FDI attacks, and establish corresponding mathematical models to describe the impact of these attacks on the system. This step provides a basis for system modeling and analysis in subsequent steps.

[0019] Step 2: Design a DoS attack defense network based on deep learning, including convolutional neural network (CNN), long short-term memory network (LSTM), and autoregressive AR module, to predict and compensate for the packet loss of system measurement values ​​caused by DoS attacks;

[0020] DoS attacks can cause the loss of system measurements, so it is necessary to design a corresponding defense mechanism to compensate for the lost data. The DoS attack defense network based on deep learning can effectively predict and compensate for the lost measurements through CNN, LSTM and AR modules, providing accurate data for subsequent state estimation and control.

[0021] Step 3: Design an adaptive FDI attack compensation state observer, adopt a compensation mechanism based on projection operator, establish compensation for unknown FDI attacks, and design an adaptive law of compensation signal to achieve the estimation of system state under network attack;

[0022] FDI attacks can tamper with system measurements, resulting in distorted state estimation. Therefore, it is necessary to design an adaptive FDI attack compensation state observer to eliminate the impact of the attack and accurately estimate the system state. The observer adopts a compensation mechanism based on the projection operator, which can effectively compensate for the errors caused by unknown FDI attacks.

[0023] Step 4: Based on the DoS and FDI attack defense mechanism, a security optimization controller is designed. An evaluation-execution structure is constructed based on reinforcement learning. The virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The neural network is used to approximate the unknown parts of the system model, evaluation network, and execution network, and a weight adjustment law is given. Considering that the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function. The properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0024] Furthermore, the DoS attack defense network based on deep learning includes:

[0025] Convolutional neural network (CNN) module, used to extract effective features of measurement data;

[0026] Long short-term memory network (LSTM) module is used to predict its long-term development trend;

[0027] Autoregressive AR module, used to improve the adaptability of deep learning model output data to changes in input data;

[0028] The CNN module first performs feature extraction on the input measurement data to extract the features in the data;

[0029] The LSTM module receives the features extracted by the CNN module and combines its own memory capacity to predict the long-term and short-term trends of the data;

[0030] The AR module receives the output of the CNN and LSTM modules, and further performs autoregressive prediction on the data to improve the model's sensitivity to the scale of the input data and adapt to the characteristics of non-periodic changes in sensor measurement data under network attacks.

[0031] Furthermore, the CNN module includes a squeeze excitation block SE-CNN for increasing the weight of feature channels useful for the current task, so that the model can adaptively adjust the feature response of the channel.

[0032] Furthermore, the adaptive FDI attack compensation state observer adopts a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks, and designs an adaptive law of the compensation signal to achieve estimation of the system state under network attacks.

[0033] Furthermore, the safety optimization controller constructs an evaluation-execution structure based on reinforcement learning, designs a virtual optimal controller and an actual optimal controller in combination with the backstepping method, uses neural networks to approximate the unknown parts of the system model, the evaluation network and the execution network, and gives a weight adjustment law. Considering the situation where the system output has a symmetric time-invariant constraint, a log-type barrier function is introduced into the performance index function, and the properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0034] The beneficial effects of the present invention are:

[0035] (1) In order to cope with the challenges brought by the openness of the communication network information transmission process to the system information security, this patent considers the secondary voltage control system of the microgrid under hybrid network attack. The DoS attack and FDI attack occur in the feedback channel between the output sensor and the controller. Under the DoS attack, the sensor detection information will be lost, which will destroy the availability security requirements of the smart grid. Under the FDI attack, false data information will be injected into the feedback channel, which will destroy the integrity security requirements of the smart grid. These two network attacks are modeled, considering their impact on the output variables of the controlled system, and the secondary voltage control model of the microgrid under hybrid network attack is derived.

[0036] (2) A prediction compensation model based on convolutional neural network and long short-term memory neural network is proposed to solve the problem of system measurement value loss caused by DoS attacks. The model can extract effective features of measurement data, predict its long-term development trend, embed the squeeze excitation block into CNN, increase the weight of feature channels that are useful for the current task, and enable the model to adaptively adjust the feature response of the channel. Compensation for missing measurements can be regarded as a time series prediction problem. Combined with the autoregressive module, it can improve the adaptability of the output data of the deep learning model to changes in the input data, and improve the accuracy and robustness of the predicted output.

[0037] (3) Design a security observer with adaptive compensation for FDI attacks to realize the state observation of the system under hybrid network attacks. FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the measurement information of the power system, thereby affecting the effect of the power control system and threatening the safe and stable operation of the power grid. It is highly concealed and destructive. The FDI attack is parameterized, and a compensation mechanism based on the projection operator is adopted to establish compensation for unknown FDI attacks. The adaptive law of the compensation signal is designed to realize the estimation of the system state under network attacks.

[0038] (4) Based on reinforcement learning, an evaluation-execution structure is constructed, and the virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The universal approximation characteristics of neural networks are used to approximate the unknown parts of the system model, evaluation network, and execution network, and the weight adjustment law is given. Considering the situation where the system output has a symmetric time-invariant constraint, a log-type barrier function is introduced into the performance index function. Using the properties of the logarithmic function, the output is constrained within a fixed constant interval to meet the constraint conditions.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 The structural block diagram of the secondary voltage control system under hybrid network attack;

[0042] Figure 2 This is the block diagram of the microgrid secondary voltage control scheme;

[0043] Figure 3 Defending deep learning networks against DoS attacks. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0046] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] In the island mode, the microgrid consists of two cascaded internal proportional-integral controllers and an external droop controller to form the primary control layer to ensure the stability of frequency and voltage. However, the frequency and voltage will inevitably deviate from the rated value, and a secondary controller needs to be designed to compensate for the deviation.

[0048] To avoid or reduce the chattering problem, a continuous control strategy can be implemented based on the Lyapunov function. The distributed secondary controller proposed based on the precise microgrid system model does not consider the system uncertainty caused by system parameter disturbances, unmeasurable variables and external load changes. In practice, measures need to be taken to prevent the system uncertainty from reducing or even destroying the stability of the microgrid. Reinforcement learning, as a machine learning method, can well solve the optimization control method of complex systems without knowing the precise system dynamics.

[0049] This patent designs an intelligent and safe microgrid secondary voltage control scheme, considers the microgrid secondary voltage control system under hybrid network attack, designs the corresponding deep learning defense network and attack compensation state observer, and meets the safety availability and integrity requirements during information transmission. Considering the system output is subject to security constraints, as well as the model uncertainty caused by system parameter disturbances and external loads, an intelligent safety controller is designed based on a neural network and barrier function combined with an evaluation-execution reinforcement learning structure.

[0050] In order to cope with the challenges brought by the openness of the communication network information transmission process to the system information security, this patent considers the secondary voltage control system of the microgrid under hybrid network attack. Among them, DoS attack and FDI attack occur in the feedback channel between the output sensor and the controller. Under DoS attack, the sensor detection information is lost, which destroys the availability security requirements of the smart grid. Under FDI attack, false data information is injected into the feedback channel, which destroys the integrity security requirements of the smart grid. Modeling these two network attacks, considering their impact on the output variables of the controlled system, derives the secondary voltage control model of the microgrid under hybrid network attack.

[0051] A prediction compensation model based on convolutional neural network and long short-term memory neural network is proposed to solve the problem of system measurement value loss caused by DoS attack. The model can extract effective features of measurement data and predict its long-term development trend. The squeeze excitation block is embedded into CNN to increase the weight of feature channels that are useful for the current task, so that the model can adaptively adjust the characteristic response of the channel. The compensation for missing measurement values ​​can be regarded as a time series prediction problem. Combining with the autoregressive module can improve the adaptability of the output data of the deep learning model to the changes in the input data, and improve the accuracy and robustness of the predicted output.

[0052] Design a security observer with adaptive compensation for FDI attacks to realize the state observation of the system under hybrid network attacks. FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the measurement information of the power system, thereby affecting the effect of the power control system and threatening the safe and stable operation of the power grid. It is highly concealed and destructive. Parameterize the FDI attack, adopt a compensation mechanism based on the projection operator, establish compensation for unknown FDI attacks, design an adaptive law of the compensation signal, and realize the estimation of the system state under network attacks.

[0053] Based on reinforcement learning, the evaluation-execution structure is constructed, and the virtual optimal controller and the actual optimal controller are designed by combining the backstepping method. The universal approximation characteristics of neural networks are used to approximate the unknown parts of the system model, evaluation network and execution network, and the weight adjustment law is given. Considering the situation where the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function, and the properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0054] This patent designs an intelligent and safe microgrid secondary voltage control scheme, considers the microgrid secondary voltage control system under hybrid network attack, designs the corresponding deep learning defense network and attack compensation state observer to meet the security and integrity requirements during information transmission. Considering the system output is subject to security constraints, as well as the model uncertainty caused by system parameter disturbances and external loads, an intelligent safety controller is designed based on a neural network and barrier function combined with an evaluation-execution reinforcement learning structure.

[0055] In order to cope with the challenges brought by the openness of the communication network information transmission process to the system information security, this patent considers the secondary voltage control system of the microgrid under hybrid network attack. Among them, DoS attack and FDI attack occur in the feedback channel between the output sensor and the controller. Under DoS attack, the sensor detection information is lost, which destroys the availability security requirements of the smart grid. Under FDI attack, false data information is injected into the feedback channel, which destroys the integrity security requirements of the smart grid. Modeling these two network attacks, considering their impact on the output variables of the controlled system, derives the secondary voltage control model of the microgrid under hybrid network attack.

[0056] A prediction compensation model based on convolutional neural network and long short-term memory neural network is proposed to solve the problem of system measurement value loss caused by DoS attack. The model can extract effective features of measurement data, predict its long-term development trend, embed the squeeze excitation block into CNN, increase the weight of feature channels useful for the current task, and enable the model to adaptively adjust the feature response of the channel. Compensation for missing measurements can be regarded as a time series prediction problem. Combining with the autoregressive module can improve the adaptability of the output data of the deep learning model to the changes in the input data, and improve the accuracy and robustness of the predicted output.

[0057] Design a security observer with adaptive compensation for FDI attacks to realize the state observation of the system under hybrid network attacks. FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the measurement information of the power system, thereby affecting the effect of the power control system and threatening the safe and stable operation of the power grid. It is highly concealed and destructive. Parameterize the FDI attack, adopt a compensation mechanism based on the projection operator, establish compensation for unknown FDI attacks, design an adaptive law of the compensation signal, and realize the estimation of the system state under network attacks.

[0058] Based on reinforcement learning, the evaluation-execution structure is constructed, and the virtual optimal controller and the actual optimal controller are designed by combining the backstepping method. The universal approximation characteristics of neural networks are used to approximate the unknown parts of the system model, evaluation network and execution network, and the weight adjustment law is given. Considering the situation where the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function, and the properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

[0059] 1. Establishment of secondary control model of microgrid under hybrid network attack

[0060] 1.1 Communication Network

[0061] Consider N distributed generation units, and the communication between them is described by an undirected graph G = (v, ε). All distributed generation units correspond to the node set v = {1, 2, ..., N}, and the communication relationship between distributed generation units is abstracted as the edges in the graph, forming the set of edges (v i ,v j ) represents the edge between nodes i and j. If v i You can j Send information, then (v i ,v j )∈ε. A=[a ij ] N×N is called the adjacency matrix of the graph. i ,v j )∈ε ij =a ji =1. Otherwise a ij =a ji =0, and a ii = 0. Define the in-degree matrix D = diag{d 1 ,d 2 ,...,d N},middle The communication network between distributed generation units is undirected and connected, and there is a path between any two nodes. Assume that the virtual leader is node υ 0 , and define the leader adjacency matrix B = diag{b 1 ,b 2 ,...,b N}, if node v 0 With node v i If there is an edge between them, then b i =1, otherwise b i =0.

[0062] 1.2 Description of Hybrid Attack

[0063] While communication networks bring great convenience, the openness of information transmission also brings unprecedented challenges to the information security of the system. Typical network attacks include denial of service attacks (DoS) and spoofing attacks. DoS attacks block communication between devices by interfering with network channels or attacking routing protocols. Spoofing attacks damage data by modifying the information of transmission packets in network channels. They mainly include false data injection (FDI) and replay attacks. FDI attacks damage the authenticity of data by injecting false information.

[0064] Considering that DoS attacks and FDI attacks occur in the feedback channel between the output sensor and the controller, the rest of the control system channels are secure and reliable network channels. DoS attacks exhaust network resources and interrupt network communications, making it impossible for sensor measurement information to be transmitted to the control center normally, causing the power grid to be paralyzed. FDI attacks are hidden and tamper with transmission information through careful design, reducing the economic operation efficiency of the power grid.

[0065] When a DoS attack occurs, the input signal in the communication network will be blocked and lost, affecting the observation effect of the safety observer and the control performance of the controller. DoS attacks attempt to block the transmission channel, resulting in data not being able to reach the controller at a specific time. Blocking the interconnection of communication modules will cause data transmission failures and even make the system unstable.

[0066] Define a time-varying parameter λ i ∈{0,1} to indicate whether the data packet is lost, then the attacked control output signal can be expressed as According to the packet loss situation, a random process is used to model it. The probability distribution is obtained as follows:

[0067]

[0068] Where λ∈[0,1] represents the packet arrival rate, so we can get λ i The expectation of i )=λ.

[0069] During data transmission, the packet arrival rate of error-free data packets depends on the signal-to-noise ratio of the transmission signal at the receiving end. When a malicious DoS attack occurs, the signal-to-noise ratio of the communication channel can be expressed as:

[0070]

[0071] Where m 1 ∈(0,1] is the channel gain between the transmitter and the receiver, m 2∈(0,1] is the attacker’s interference gain, P is the transmitter’s transmission power, J is the attacker’s interference power, and N is the background noise. Based on the signal-to-noise ratio of the communication channel under DoS attack, the packet arrival rate can be expressed as:

[0072]

[0073] in c>0 is a constant.

[0074] Considering that FDI attacks can cause system performance degradation or even instability by destroying data accuracy, this patent is dedicated to designing secure observers and controllers to ensure the stability of networked control systems even when FDI attacks occur. When an FDI attack occurs, false information will be injected into the system's output signal, causing data accuracy to be destroyed. The output of the attack is represented by in is the output variable under FDI attack, a i It is a signal of FDI attack.

[0075] 1.3 System model establishment

[0076] Considering the situation where the secondary control system of the microgrid is simultaneously attacked by DoS and FDI hybrid networks, and considering the security of the system control process, the system output is constrained. Microgrids lack the inertia of traditional power grid generators and have disadvantages such as low inertia, variable network structure, and parameter uncertainty. At the same time, the output of most distributed power generation units is volatile and intermittent due to environmental factors, which poses a huge challenge to maintaining the stability of the microgrid system. In summary, considering the uncertainty of parameter disturbances, unknown parameters, and unmeasurable variables that may occur in the controlled object, the secondary voltage control system of the microgrid with output constraints under hybrid network attacks is modeled.

[0077] The block diagram of the microgrid control system under the hybrid network attack of DoS and FDI is shown in the figure. The primary control layer consists of two cascaded internal proportional integral controllers and an external droop controller to ensure the stability of frequency and voltage. However, the frequency and voltage will inevitably deviate from the rated value, so the secondary controller is designed to compensate for the deviation value. Droop control can achieve power self-consistency between distributed generators through the output of different distributed generator sets. As an effective distributed control method that does not rely on the communication network, it is widely used in microgrids.

[0078] Microgrid model Figure 1 As shown, each distributed generation unit consists of a three-phase inverter with a DC power supply, an inductor-capacitor filter L fi and C fi , output connector R ci and Lci , and the interconnection link impedance R li and L li The controller is designed on the two-phase rotating dq coordinate system. The controller consists of a two-layer structure, including a primary controller and a secondary controller. This is the system output when it is attacked by a combination of DoS and FDI.

[0079] The function of the primary controller is to keep the output voltage and frequency stable within a preset range. However, as the load conditions of the microgrid change, the primary controller cannot prevent the frequency and voltage from deviating from the rated values. Therefore, a secondary controller is required to compensate for the deviations in voltage and frequency.

[0080] The primary controller includes external droop control and internal voltage and current control. The droop control indicates the reference frequency and reference voltage of the i-th distributed generation unit. The voltage and current controllers enable the three-phase inverter to track the reference frequency and voltage values ​​provided by the droop controller.

[0081] The dynamic equation of the power control module using droop control can be expressed as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Where P i and Q i Indicates the cutoff frequency is ω ci Instantaneous active power and reactive power of the low-pass filter, and is the droop coefficient of frequency and voltage in primary droop control, v ni and ω ni are the expected voltage and frequency of the ith generating unit, and for The dq component, v odi and v oqi andi odi and i oqi are the output voltage and output current of the i-th power generation unit in the dq reference coordinate system. The time-varying parameter λ i ∈{0,1} to indicate whether the data packet is lost when attacked by DoS, ai (t) is the FDI attack signal, i.e., the false information injected into the output signal.

[0088] The control of voltage and current is usually implemented using the PI control method, giving the differential equations of the voltage and current controllers:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] where i ldi ,i lqi ,v ldi ,v lqi i li and v li The d-axis and q-axis components, K PVi ,K IVi ,K PCi ,K ICi are the proportional and integral gains for voltage and current control, respectively, ω b is the nominal angular frequency, F i is a constant gain value.

[0096] The dynamic equations of the output LC filter and output connector are described by the following differential equations.

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] where i ldi and i lqi and v bdi and v bqi They are the inverter output current i li and bus voltage v biThe dq components of .

[0104] The output right Differentiate to get the output voltage of the voltage source inverter The input voltage of the droop controller is ni The relationship between them is as follows:

[0105]

[0106] Simplifying the large signal model into a second-order system, the system state equation of secondary voltage control under hybrid network attack is as follows:

[0107]

[0108]

[0109]

[0110] The control input signal is v ni , is the output of the DoS and FDI hybrid network attack, λ i ∈{0,1} represents the loss of data packets under DoS attack, a i It is a signal of FDI attack. is the output variable of the controlled system under hybrid attack, function f i and g i as follows:

[0111]

[0112]

[0113] Parameter disturbances, unknown parameters and unmeasured variables of the controlled system will cause f i Uncertain coefficients cause deviations in the system dynamic model and reduce control performance. Circuit parameter changes, such as the inductor capacitor filter L fi and C fi In practical applications, it is difficult to obtain the output connector parameter R ci , L ci , and the voltage variable v bdi , there are unknown parameters and unmeasured variables. In particular, the inductance and capacitance parameters are of small magnitude, resulting in f i The coefficient disturbance deviation is large. Due to the existence of uncertain factors and interference, such as the voltage fluctuation on the public line caused by the change of local load in the system, ξ i Represents the uncertainty of the system model caused by parameter disturbance.

[0114] 2. Design of Intelligent Safety Secondary Voltage Controller

[0115] In summary, considering that the controlled object is subject to a hybrid network attack of DoS and FDI, and considering that the system has output constraints, and that the controlled object may have uncertainties such as parameter disturbances, unknown parameters and unmeasurable variables, an intelligent and safe controller is designed for the microgrid secondary control system. The design scheme is as follows Figure 2 shown.

[0116] 2.1 DoS attack defense network based on deep learning

[0117] DoS attacks transmit destructive information to the power system's communication network, exhaust network resources, interrupt network communications, and prevent the control center from receiving system measurement data. DoS attackers launch malicious attacks to prevent the system control center from obtaining and using system measurement data in a timely and reliable manner. In real life, the disastrous consequence is cascading power outages, affecting the stability of the power grid and causing economic losses.

[0118] DoS attacks disrupt the communication channels of the controlled systems, resulting in the loss of system output sensor measurement data transmitted to the secondary control center. For power grids that rely on measurement data for state estimation, the loss of measurement data will reduce the accuracy of the estimation. Considering the dynamic nonlinear characteristics of the power system and the effectiveness of deep neural networks in fitting nonlinear functions, deep learning is used to predict and compensate for the lost measurement values.

[0119] A hybrid neural network model based on deep learning, consisting of CNN, LSTM and AR modules, is used to predict and compensate for the measurement loss of the power system state under DoS attacks. The network structure Figure 3 shown.

[0120] SE-CNN adds a squeeze excitation block to CNN to increase the weight of feature channels that are useful for the current task. The CNN module and LSTM module are used to extract short-term local features between measurements and predict long-term development trends. The AR model is used to improve the sensitivity of the neural network model to the scale of input data, to cope with the characteristics of non-periodic changes in sensor measurement data under network attacks, and to improve the prediction effect. Each module of the hybrid neural network simultaneously receives the same system measurement value as input and calculates the result, uses the Concat function to concatenate the features of the result, and then obtains the predicted system measurement value through the fully connected layer. Finally, the predicted value replaces the missing data value in the actual measurement value, thereby realizing the compensation process.

[0121] Assume that the actual measurement value is Y. When the system is attacked by DoS, the system measurement value will be randomly lost during the transmission process, and the measurement value collected by the control center is Y. r , need to compensate Yr To ensure the accuracy of the state estimation result, the missing measurement values ​​in the compensation matrix are defined as The objective of measurement value compensation is expressed as follows, where Θ is the set of model parameters:

[0122]

[0123] The calculation of each time step of LSTM is as follows:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] Where σ is the sigmoid function, tanh is the hyperbolic tangent function, the symbol * is the Hadamard multiplication, W is the weight matrix, b is the bias, and g t is the activation vector. The LSTM model is composed of the input word v t , cell status Temporary cell state Hidden state Forget Gate Memory Gate and output gate The calculation process is to forget the old information in the cell state and remember the new information, so that the information useful for subsequent calculations can be transmitted. And the hidden state is output at each time step The forgetting, memory and output processes are controlled by the hidden state of the previous moment. and the current input v t The calculated forget gate, memory gate and output gate are used to control it.

[0131] A single LSTM network cannot fully meet the prediction requirements of multivariate and complex time series scenarios. The SE-CNN module is added to extract the features of multiple complex time series and improve the prediction performance. The module is composed of three stacked convolutional blocks with skip-layer connection structures. The first and second convolutional blocks include a convolutional layer, a batch normalization layer and an SE layer, and the third convolutional block includes a convolutional layer and a batch normalization layer. The SE block is equivalent to adding an attention mechanism to the channel dimension, which is crucial to improving the performance of the model in dealing with complex data sets. Through the fully connected network, a new neural network is used to obtain the importance of each channel of the feature map according to the automatic learning method of the loss, and then a weight value is assigned to each feature, so that the importance of each feature channel changes, and the neural network focuses on the channels with large weight values.

[0132] F tr As a convolution operator, its input is X∈R W′×H′×C′ , the output is U∈R W×H×C , expressed as U = [u 1 ,u 2 ,...,u cf ], the convolution process can be expressed as

[0133]

[0134] Where * represents the convolution operation, is a two-dimensional space kernel, representing v cf A single channel of acts on the corresponding channel of X. Next, the weights of channel features are adaptively adjusted through squeezing, excitation, and scaling.

[0135] The squeezing operation obtains global information through a global average filter. The time dimension feature K of each channel time series data is pooled by the global average pooling F sq Compressed into a real number, this real number is the statistics z of the channel dimension, and the cth element of z is z cf :

[0136]

[0137] The excitation operation implements the function of adaptive recalibration, using a Bottleneck structure consisting of two fully connected layers to simulate the correlation between channels and output the same number of weight values ​​s as the input features:

[0138] s=F ex (z,W)=σ(g(z,W))=σ(W 2 δ(z,W 1 z)) (2.10)

[0139] Where δ represents the ReLU function, W1 and W 2 They are the parameters of the dimension reduction layer and the dimension increase layer respectively. This part first compresses the features through a fully connected layer, then uses the ReLU layer for nonlinear operation, then uses the fully connected layer to restore the features, and finally uses the sigmoid function for activation.

[0140] Finally, s is multiplied by the previous convolution layer in spatial position, and the normalized weight is weighted to the features of each channel. Finally, the output of this module is:

[0141]

[0142] Among them, F scale (u cf ,s cf ) represents the channel multiplication between the feature map ucf and the scale scf,

[0143] The sensor measurement data under cyber attack changes continuously with non-periodic characteristics. The LSTM and SE-CNN models are insensitive to changes in the input scale. Adding an autoregressive module improves the prediction effect. The AR model formula is:

[0144]

[0145] in is the prediction result of AR module, W a is the coefficient of the AR module, q a is the size of the input window.

[0146] The LSTM module, SE-CNN module, and AR module simultaneously receive the same measurement input and calculate their respective results, use the Concat function to concatenate the features of the results, and then obtain the prediction results through the fully connected layer.

[0147] The actual measurement value sent at each moment is defined as Y, and the attack matrix is ​​represented by A = random (Y, δ). The measurement value received by the control center is expressed as:

[0148] Y r =A.*Y (2.13)

[0149] A contains only two values: 0 and 1. When it is 0, it means that it is under attack and the measurement data will be lost in the data received by the control center.

[0150] Y r Input the hybrid neural network model to get the predicted value of the measured value

[0151]

[0152] The missing measurements are replaced with the predicted values ​​and combined with the non-missing measurements to form the final compensation matrix Y:

[0153]

[0154]

[0155] Compensation measurement value Input to the central estimator to estimate the state of the power system.

[0156] Output voltage odi When the sensor detects that the data is attacked by DoS during transmission to the secondary control center There will be data loss, and the compensation measurement value obtained by using the DoS attack defense network is:

[0157]

[0158] Output of a hybrid network attack After using the deep learning defense network to predict and compensate for the missing data values, the output is

[0159] 2.2 Adaptive FDI attack compensation state observer

[0160] FDI attacks can successfully bypass the bad data detection mechanism in the power grid by tampering with the power system measurement information, thereby affecting the power system state estimation results and threatening the safe and stable operation of the power grid. It is highly concealed and destructive. After passing through the DoS attack defense network, the voltage input to the control center is

[0161] a i The FDI attack on the system can be parameterized as:

[0162]

[0163] The unknown time-varying gain w i (t)≠-1, and satisfies and Are all unknown positive real numbers, define the function:

[0164]

[0165] Among them sup t∈[0,∞) |ψ i |<∞, The system output has the following relationship:

[0166]

[0167] By adopting the compensation mechanism based on projection operator, designing adaptive attack compensation observer and adaptive law of compensation signal, the estimation of system state under network attack is realized. Attack compensation for unknown FDI attack is established so that the compensation state converges to the real system state.

[0168] The system state observer is designed as:

[0169]

[0170] Among them l i,1 , l i,2 are the parameters of the state observer, It is right i The estimate, Estimated output value of the hybrid network attack is the compensation measurement value output by the DoS attack defense network. i There is uncertainty under the influence of factors such as parameter disturbance and external load changes, and the neural network structure can be used for approximation:

[0171]

[0172] use approximate And the regulation law design The design parameter γ f,i >0,σ i >0,z i,2 is the error value between the state and the virtual controller. The specific design method is given in the next section.

[0173] The designed state observer has the ability to compensate for FDI network attacks, where the compensation signal adaptive law The design of affects the system observation capability and tracking performance. The adaptive law is designed as follows:

[0174]

[0175] in is a compensation signal applied to FDI attacks The projection norm bound on , the projection operator m i is a positive real number. q i >0,L i =[l i,1 l i,2 ] T .

[0176] Introduce the projection operator to transform the parameter ψ i It is restricted to the set convex set to ensure uniform boundedness. Therefore, the attack compensation observer is designed based on the projection operator, which can estimate the unknown time-varying gain generated by network attacks.

[0177] 2.3 Safety optimization controller design

[0178] Combining the knowledge of reinforcement learning and neural networks, an optimized coordination controller is designed for the microgrid secondary voltage control multi-agent system based on the optimized backstepping controller design method to achieve rated voltage tracking and minimize the performance indicator function.

[0179] Defining the Error Signal is the optimal virtual control signal.

[0180] first step:

[0181] The tracking error signal is derived as follows:

[0182]

[0183] Define consistent tracking error:

[0184]

[0185] Where N i is the neighbor node of the i-th agent, and the dynamic equation of the consistency error is:

[0186]

[0187] in

[0188] Considering the symmetric time-invariant output constraints k ci is a known positive real number, then the consistent tracking error has the constraint -k bi <e i <k bi , the optimal performance indicator function is defined as follows. A log-type barrier function is introduced into the performance indicator to ensure that the system tracking error does not exceed the constraint interval.

[0189]

[0190] Among them, the parameter h i >0, As the optimal virtual control Taking the time derivative of both sides of the performance index function, the HJB equation is obtained:

[0191]

[0192] Find the partial derivative of the HJB equation to the optimal virtual controller, that is have to:

[0193]

[0194] Will This breaks down into:

[0195]

[0196] where β i is a positive real number of design, and Will Substituting into the optimal virtual control law expression, we get:

[0197]

[0198] is an unknown and continuous function, which is approximated by a neural network and is expressed as follows:

[0199]

[0200] in is the ideal neural network weight, q is the number of neurons, is the basis function vector, is the approximation error. Since the weight matrix is unknown, so the optimal virtual control law is not available. Based on the reinforcement learning method, the evaluation network and the execution network are constructed to derive the available control law.

[0201] The evaluation network designed to evaluate the controller performance is expressed as follows:

[0202]

[0203] in Yes The estimate, Is the evaluation network weight, the designed update law is as follows:

[0204]

[0205] The evaluation network design parameter γ c,i1 >0.

[0206] The action network designed for execution control is as follows:

[0207]

[0208] in is the action network weight, and its update law is as follows:

[0209]

[0210] The action network design parameter γ a,i1 >0.

[0211] Will Substituting into the HJB equation, we get:

[0212]

[0213] like holds and has a unique solution, which is equivalent to:

[0214]

[0215] Weight value According to the update law, it is continuously adjusted until Finally, the optimal virtual control is obtained.

[0216] Step 2:

[0217] definition The derivative of the error dynamic equation is:

[0218]

[0219] Define the optimal performance indicator function:

[0220]

[0221] in, For the optimal actual controller, the time derivatives of both sides of the performance index function are obtained, and the HJB equation is obtained as follows:

[0222]

[0223] Find the partial derivative of the HJB equation to the optimal virtual controller, that is have to:

[0224]

[0225] Will Decompose into the following form:

[0226]

[0227] where γ i is a positive real number of design, and Will Substituting into the optimal control law expression, we get:

[0228]

[0229] f i and is an unknown and continuous function, approximated by a neural network, and its expression is as follows:

[0230]

[0231] in and is the ideal neural network weight, q is the number of neurons, S f,i and is the basis function vector, ε f,i and is the approximation error. Since the weight matrix and is unknown, so the optimal control law is not available. Based on the reinforcement learning method, an evaluation network and an execution network are constructed to derive an available control law.

[0232] Design the evaluation network for evaluating control performance and the action network for executing control as follows:

[0233]

[0234] in Yes The estimate, is the identification network weight, is the evaluation network weight, is the action network weight, and the update law is designed as follows:

[0235]

[0236]

[0237]

[0238] The identification network design parameter γ f,i >0,σ i >0, evaluate network design parameter γ c,i2 > 0, action network design parameter γ a,i2 >0.

[0239] At this point, the design of the microgrid voltage secondary controller under hybrid network attack is completed. The algorithm flow is shown in the following pseudo code:

[0240]

[0241]

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A microgrid intelligent safety secondary voltage control method based on reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Establish a microgrid secondary control model under hybrid network attacks, including: Establish a communication network model, in which the communication relationship between distributed generation units is abstracted as an undirected graph; Model DoS attacks and FDI attacks, and establish a packet loss probability distribution model and an FDI attack signal model respectively; The state equation of the microgrid secondary voltage control system with output constraints under hybrid network attacks is established, taking into account uncertainty factors such as system parameter disturbances, unknown parameters and unmeasurable variables; Step 2: Design a DoS attack defense network based on deep learning, including convolutional neural network (CNN), long short-term memory (LSTM) network and autoregressive (AR) module, to predict and compensate for the packet loss of system measurement values ​​caused by DoS attacks; Step 3: Design an adaptive FDI attack compensation state observer, adopt a compensation mechanism based on projection operator, establish compensation for unknown FDI attacks, and design an adaptive law of compensation signal to achieve the estimation of system state under network attack; Step 4: Based on the DoS and FDI attack defense mechanism, a security optimization controller is designed. An evaluation-execution structure is constructed based on reinforcement learning. The virtual optimal controller and the actual optimal controller are designed in combination with the backstepping method. The neural network is used to approximate the unknown parts of the system model, evaluation network, and execution network, and a weight adjustment law is given. Considering that the system output has symmetric time-invariant constraints, a log-type barrier function is introduced in the performance index function. The properties of the logarithmic function are used to constrain the output to a fixed constant interval to meet the constraints.

2. According to a microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1, it is characterized by: The DoS attack defense network based on deep learning includes: Convolutional neural network (CNN) module, used to extract effective features of measurement data; Long short-term memory network LSTM module, used to predict its long-term development trend; Autoregressive AR module, used to improve the adaptability of deep learning model output data to changes in input data; The CNN module first performs feature extraction on the input measurement data to extract the features in the data; The LSTM module receives the features extracted by the CNN module and combines its own memory capacity to predict the long-term and short-term trends of the data; The AR module receives the output of the CNN and LSTM modules, and further performs autoregressive prediction on the data to improve the model's sensitivity to the scale of the input data and adapt to the characteristics of non-periodic changes in sensor measurement data affected by network attacks.

3. According to a microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 2, it is characterized in that: The CNN module includes a squeeze excitation block SE-CNN for increasing the weight of feature channels useful for the current task, so that the model can adaptively adjust the feature response of the channel.

4. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1 is characterized in that: The adaptive FDI attack compensation state observer adopts a compensation mechanism based on a projection operator to establish compensation for unknown FDI attacks, and designs an adaptive law of compensation signals to achieve estimation of the system state under network attacks.

5. The microgrid intelligent safety secondary voltage control method based on reinforcement learning according to claim 1 is characterized in that: The safety optimization controller constructs an evaluation-execution structure based on reinforcement learning, designs a virtual optimal controller and an actual optimal controller in combination with the backstepping method, uses a neural network to approximate the unknown parts of the system model, the evaluation network and the execution network, and gives a weight adjustment law. Considering that the system output has a symmetric time-invariant constraint, a log-type barrier function is introduced into the performance index function, and the properties of the logarithmic function are used to constrain the output within a fixed constant interval to meet the constraint conditions.

Images