A new energy micro-grid secondary control method, device, equipment and medium

Abstract

The application discloses a new energy micro-grid secondary control method, device, equipment and medium, relates to the safe and stable control technical field of power system, and first forms a deviation characteristic vector by calculating the deviation of a state variable and a historical mean value, and generates an adaptive weight, weights a historical state sequence, quantifies the importance of each state variable dynamically, and highlights the key event characteristics;Secondly, the weighted state sequence is input into a long short-term memory network (LSTM), the time dependence of the state data is deeply mined, and the hidden state containing the historical evolution mode is obtained;Finally, the agent controls the decision based on the hidden state through a double-delay deep deterministic policy gradient (TD3) algorithm, and determines the control action.The application utilizes the memory enhancement mechanism to identify the key historical mode, and combines reinforcement learning to make a robust decision, can efficiently and stably control the micro-grid, and the effect is remarkable especially when coping with network attacks and working condition disturbances.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system safety and stability control, and particularly relates to a new energy micro-grid secondary control method, device, equipment and medium. BACKGROUND

[0002] A micro-grid is a small power system composed of distributed power sources, energy storage devices, loads and energy conversion devices. Due to the random nature of the distributed power sources, the micro-grid control and power quality are easily adversely affected. To solve this problem, the micro-grid power quality hierarchical control improves the operation stability through multi-level regulation technology, and includes a collaborative architecture of primary control, secondary control and tertiary control. The secondary control, i.e. the secondary control, dynamically adjusts the voltage frequency, amplitude, harmonics and imbalance. At present, the micro-grid presents typical information-physical fusion characteristics due to the integration of communication and control technology, bringing more severe challenges: the secondary control needs to be safe under the double security threats of the information layer and the physical layer; the integration of communication networks in the information-physical system makes the system vulnerable to network attacks, among which the false data injection (FDI) attack is a typical attack type. Attackers tamper with sensor data or inject false signals, causing the control system to make decisions based on incorrect information, leading to system frequency and voltage instability, and even causing system collapse in severe cases. In addition, the micro-grid also faces load fluctuations, line disconnections, short circuits and other faults under different operating conditions. These faults will cause power loss and system instability, further increasing the difficulty of control strategy design.

[0003] At present, the secondary control method of the micro-grid mainly includes model-driven and data-driven methods. The model-driven method, such as proportional-integral-derivative control and model predictive control, relies on fixed physical models, but the modeling accuracy is difficult to guarantee, and it cannot adapt to the nonlinear characteristics of high proportion of new energy, resulting in insufficient adaptability and robustness. The data-driven method takes the reinforcement learning algorithm as the core, such as the deep deterministic policy gradient algorithm and the double-delay deep deterministic policy gradient algorithm, which overcomes the dependence on physical models of the model-driven method. By constructing a policy network and an evaluation network, the micro-grid secondary control problem is modeled as a Markov decision process.

[0004] However, the traditional reinforcement learning method uses a fully connected network (FCN) architecture, and its feedforward network structure lacks a memory mechanism, making it difficult to identify and utilize key historical state evolution patterns in the process of network attacks gradually penetrating and operating conditions dynamically changing. It only processes the state snapshot at the current time, and it is difficult to capture the key long-term dependencies in the state sequence, making it difficult to efficiently control the micro-grid. Therefore, there is an urgent need for a new secondary control method that can effectively utilize historical state information and has long-term memory capability to cope with increasingly complex network attacks and operating conditions. SUMMARY

[0005] The embodiment of the application provides a new energy micro-grid secondary control method, device, equipment and medium, and can solve the problems in the prior art.

[0006] The embodiment of the application provides a new energy micro-grid secondary control method, device, equipment and medium, and can solve the problems in the prior art.

[0007] The secondary control process of the new energy micro-grid is constructed as a Markov decision process, wherein a state space includes a state vector composed of physical quantities representing dynamic characteristics of the new energy micro-grid, and an action space includes control instructions representing controller outputs for secondary control of the new energy micro-grid.

[0008] A historical state sequence of a plurality of time steps in the operation process of the new energy micro-grid is collected; for each state vector in the state sequence, a state variable mean of a plurality of state variables in the state vector is obtained, a deviation of the state variable from the state variable mean is calculated, a deviation amplitude vector is formed based on a plurality of deviations, and a deviation feature vector representing the importance of the state variables in the state vector is obtained through feature transformation of the deviation amplitude vector by a linear transformation layer defined by a weight matrix and a bias vector and a nonlinear activation function.

[0009] The deviation feature vector is multiplied by a preset sensitivity parameter to obtain an adaptive amplification factor; the adaptive amplification factor is selectively applied to state variables associated with micro-grid stability in the state vector through a predefined binary mask vector to obtain an adaptive weight vector; and the state vector is multiplied by the adaptive weight vector element by element to obtain a weighted state vector and a weighted state vector sequence.

[0010] The weighted state vector sequence is input into a long short-term memory network (LSTM) to capture deep temporal dependencies of state data in the state vector sequence, and a hidden state of the weighted state vector sequence is obtained.

[0011] A control decision response is made according to the hidden state to determine a control instruction of the controller for secondary control of the new energy micro-grid.

[0012] Preferably, the deviation feature vector is obtained by:

[0013] A state sequence of a length T in the operation process of the new energy micro-grid is collected , wherein each state vector , is the dimension of the state space;

[0014] For the i-th component in the state vector , the state component mean is obtained from the past The mean value of one time step is calculated, and the formula is:

[0015] ;

[0016] The absolute difference between the current state component and the mean value of the state component is calculated to form a deviation amplitude vector The formula is:

[0017] ;

[0018] The deviation amplitude vector is transformed through a learnable linear layer and a nonlinear activation function to obtain a transformed feature vector ; and a moving average filter with a window size of is applied to smooth the transformed feature vector to obtain a smoothed deviation feature vector ; the formula is as follows:

[0019] ;

[0020] Where: j represents the dimension index of the vector.

[0021] Preferably, the weighted state vector is obtained, including:

[0022] The smoothed deviation feature vector is multiplied by a sensitivity parameter , and processed through a bounded activation function to generate a final adaptive amplification factor ; the bounded activation function limits the amplification effect within a preset maximum value , and the formula is:

[0023] ;

[0024] ;

[0025] A predefined binary mask vector is selectively applied to the state variable associated with the control;

[0026] The adaptive weight vector is obtained by adding the reference weight (all-1 vector) and the masked amplification factor; the original state vector is multiplied element by element with the adaptive weight vector to obtain a weighted state vector , and the formula is:

[0027] ;

[0028] ;

[0029] where ⊙ denotes element-wise multiplication.

[0030] Preferably, the obtaining the hidden state of the weighted state vector sequence comprises:

[0031] The weighted state vector sequence is , which is input to a long short-term memory network (LSTM);

[0032] At each time step k of the state vector sequence , a forgetting gate calculates an activation value vector between 0 and 1 based on the weighted state vector at the current time and the hidden state at the previous time , each element of the vector corresponding to an information dimension in the cell state at the previous time ; "1" means complete retention, and "0" means complete discard; the calculation formula is represented as:

[0033] ;

[0034] wherein: and represent weight matrices; represent bias vectors; and σ represents a Sigmoid activation function.

[0035] The input gate determines which values need to be updated through a Sigmoid function, and creates a new candidate value vector through a tanh layer, which contains new information that may be added to the cell state; the calculation formula is as follows:

[0036] ;

[0037] ;

[0038] wherein: , , and represent weight matrices; and represent bias vectors.

[0039] The old cell state is multiplied element-wise by the output of the forgetting gate , and the discarded information is discarded; the output of the input gate is multiplied element-wise by the candidate value vector Element-wise multiplication, get the information that needs to be added; add the two parts to get the new cell state at the current time ; The formula is:

[0040] ;

[0041] The output gate determines a filtering condition through the Sigmoid function , the updated cell state is processed through the tanh function, and is multiplied element-wise with to obtain the final hidden state output at the current time ; The formula is:

[0042] ;

[0043] .

[0044] Preferably, when responding to the control decision according to the hidden state, an actor-critic architecture based on double-delay deep deterministic policy gradient TD3 is adopted, and a reward function combined with knowledge assistance is constructed based on the initial reward function of the Markov decision process to respond to the control decision, including:

[0045] The actor network and the critic network both receive the hidden state sequence as input, perform nonlinear processing through a fully connected layer, and output a decision;

[0046] The actor network outputs a deterministic action according to the current policy function μ, and adds Gaussian noise in the training process to promote policy exploration; the action generation formula is:

[0047] ;

[0048] Where: is the parameter of the actor network;

[0049] A composite reward function with knowledge assistance is set , and the function considers multiple control objectives, and its expression is:

[0050] ;

[0051] Where each component is defined as:

[0052] : penalize the voltage deviation of the public coupling point (PCC);

[0053] : penalize voltage deviation of two energy storage units;

[0054] : penalize system frequency deviation;

[0055] : impose a binary penalty when the agent's action leads the system to extreme operating conditions;

[0056] : impose a tiny penalty proportional to the simulation time step to encourage the agent to take more efficient control actions;

[0057] : this is a cyber attack aware penalty term, whose attack scaling factor is dynamically adjusted according to attack type, taking different positive values for random signal attack (RSA), decaying signal attack (DSA) and periodic signal attack (PSA), thus increasing the concern for system stability when under attack.

[0058] Preferably, it further comprises training the constructed Markov decision process, including:

[0059] During the training process, the agent stores the experience tuples generated during the interaction process into an experience replay pool, from which small batches of data are randomly sampled for training;

[0060] The TD3 algorithm uses a pair of Critic networks ( , ) and their corresponding target networks ( , ) to suppress overestimation of Q values; the calculation of target Q value uses the Clipped Double-Q Learning method, which is expressed as:

[0061] ;

[0062] where: is the discount factor; the action is the next time action generated by the target Actor network with added clipping noise;

[0063] Each Critic network is updated by minimizing the mean squared Bellman error, and its loss function is:

[0064] ;

[0065] The Actor network is updated by policy gradient, and the output of the first Critic network is used to guide the improvement direction of the policy, and the policy gradient is approximately:

[0066] ;

[0067] The parameters of all target networks are slowly close to the current network parameters θ by a soft update mechanism, and the update formula is:

[0068] ;

[0069] Wherein: τ is a soft update coefficient much smaller than 1;

[0070] And a phased training strategy is adopted: in the initial stage, the agent is trained under various operating conditions without any network attack, after this stage is completed, the special training stage for specific attack types is entered in turn; And when entering the attack special training stage, the strength of the exploration noise is reduced to stabilize the learning process to complete the training.

[0071] Preferably, the construction of the Markov decision process comprises:

[0072] The island operation microgrid model containing synchronous generator SG, photovoltaic PV and battery energy storage system BESS is constructed, and the microgrid model secondary control problem is modeled as a Markov decision process;

[0073] Wherein, the state space S is defined as a multi-dimensional vector containing the current system frequency , the frequency deviation , the previous time frequency deviation , the point of common coupling (PCC) voltage , the PCC voltage deviation , the previous time PCC voltage deviation , the node voltage of the two energy storage units and and the voltage deviation and , and the active power input P1 and P2 of the two energy storage units.

[0074] The action space A is defined as a four-dimensional control vector output by the controller, which respectively represents the active power and reactive power input for controlling the two energy storage units.

[0075] The embodiment of the application also provides a new energy microgrid secondary control device, comprising:

[0076] An agent module is configured to construct a secondary control process of the new energy micro-grid as a Markov decision process, wherein a state space includes a state vector composed of physical quantities representing dynamic characteristics of the new energy micro-grid, and an action space includes control instructions representing controller outputs for secondary control of the new energy micro-grid.

[0077] A situation awareness module is configured to collect a historical state sequence of multiple time steps in a running process of the new energy micro-grid, obtain a state variable mean of multiple state variables in each state vector in the state sequence, calculate a deviation of the state variable from the state variable mean, form a deviation amplitude vector based on multiple deviations, perform feature transformation on the deviation amplitude vector through a linear transformation layer defined by a weight matrix and a bias vector and a nonlinear activation function, and obtain a deviation feature vector representing an importance degree of the state variable in the state vector.

[0078] The deviation feature vector is multiplied by a preset sensitivity parameter to obtain an adaptive amplification factor, the adaptive amplification factor is selectively applied to state variables associated with micro-grid stability in the state vector through a predefined binary mask vector to obtain an adaptive weight vector, and the state vector is multiplied by the adaptive weight vector element by element to obtain a weighted state vector and a weighted state vector sequence.

[0079] The weighted state vector sequence is input into a long short-term memory (LSTM) network to capture deep-level time dependencies of state data in the state vector sequence, and a hidden state of the weighted state vector sequence is obtained.

[0080] A decision control module is configured to make a control decision response based on the hidden state, determine a control instruction of the controller, and perform secondary control on the new energy micro-grid.

[0081] Embodiments of the present application also provide an electronic device including a memory and a processor.

[0082] The memory is configured to store a computer program.

[0083] The processor is configured to execute the computer program stored in the memory to implement the steps of the new energy micro-grid secondary control method.

[0084] Embodiments of the present application also provide a computer-readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the steps of the new energy micro-grid secondary control method.

[0085] Embodiments of the present application provide a new energy micro-grid secondary control method, device, equipment and medium, which have the following advantages compared with the prior art:

[0086] The application firstly collects a historical state sequence to obtain a deviation feature vector representing the importance of state variables in a state vector, and the deviation feature vector is obtained by using historical state data of the micro-grid and an adaptive amplification factor, the adaptive amplification factor is applied to state variables in the state vector associated with micro-grid control, the state vector is adaptively weighted to dynamically evaluate and quantify the importance of each state variable in the state sequence, and the specific method is to solve the problem in the state processing homogenization in the traditional method through a state adaptive input weighting mechanism, to identify the key historical state evolution mode and provide the necessary premise for subsequent use, and then the weighted state vector sequence is input into a long short-term memory network (LSTM) to capture the deep time dependence of the state data in the state vector sequence, so as to capture the long-term dependence of the key historical state evolution mode in the state sequence, and to efficiently perform secondary control on the micro-grid. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 A whole flowchart of a new energy micro-grid secondary control method provided by the embodiment of the application is shown in the figure.

[0088] Figure 2 A reward evolution diagram in a phase reinforcement learning training process of a new energy micro-grid secondary control method provided by the embodiment of the application is shown in the figure.

[0089] Figure 3 A whole flowchart of a new energy micro-grid secondary control method provided by the embodiment of the application is shown in the figure.

[0090] Figure 4 A whole flowchart of a new energy micro-grid secondary control method provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0091] In order to make the above objectives, features and advantages of the application more apparent, specific embodiments of the application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a comprehensive understanding of the application. However, the application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the application, so the application is not limited to the specific embodiments disclosed below.

[0092] At present, the microgrid secondary control method using traditional reinforcement learning algorithm, especially the control scheme based on deep deterministic policy gradient (DDPG) algorithm and double-delay deep deterministic policy gradient (TD3) algorithm; the existing traditional TD3 control scheme models the microgrid secondary control problem as a Markov decision process by constructing a reinforcement learning framework containing a policy network and an evaluation network; this scheme directly inputs the system state at the current time into a fully connected neural network, generates control actions through the policy network, and stores historical interaction data in the experience replay buffer for network training; during the training process, different network attacks and operating conditions are randomly injected, and it is expected that the agent will learn a robust control strategy through a large number of trial and error.

[0093] The existing traditional reinforcement learning method uses a fully connected network (FCN) architecture, which only processes the state snapshot at the current time and cannot capture the long-term dependencies of the state sequence; the fundamental flaw of this architecture design is that its feedforward network structure lacks a memory mechanism, which makes it impossible for the system to identify and utilize historical state evolution patterns during the gradual penetration of network attacks and dynamic changes in operating conditions; experimental data show that in complex long-sequence related scenarios, the cumulative return value of the traditional FCN architecture is significantly lower than that of a network architecture with a memory mechanism, with slow convergence speed and unstable performance. The traditional method uses a unified processing method for all state components, lacking a dynamic weight adjustment mechanism; the root cause of this homogenization problem is the failure to distinguish the importance differences of different state variables at a particular time; when critical events (such as a sharp voltage drop or frequency oscillation) occur, the system cannot highlight these key change information; therefore, when faced with multi-mode network attacks or sudden operating faults, the response accuracy and speed of the controller are severely limited; the existing technology uses a random mixed training method, introducing various network attacks and operating conditions into the training environment at the same time; the core problem of this training strategy is that event overlap causes reward signal ambiguity—when multiple disturbances occur simultaneously, the agent cannot accurately determine which factor caused the performance decline, resulting in incorrect policy updates; the traditional method uses a general reward function and does not design a differentiated reward mechanism for different types of network attacks; this design flaw stems from a lack of understanding of network attack diversity, treating all abnormal situations equally; due to the lack of attack-aware reward adjustment factors, the system uses the same optimization objective when facing attacks of different severity and types, resulting in a lack of targeted control strategy.

[0094] Based on the existing problems in the secondary control method of the microgrid at the present stage, the network structure with a memory mechanism is introduced, the modeling ability of long-term dependence relationship of state sequence is enhanced, so that the stable and accurate control performance is maintained under the gradual attack and dynamic working condition; the dynamic weight adjustment mechanism is constructed, which can distinguish the importance of different state variables, highlight the key event characteristics and improve the rapid response ability of the controller to sudden disturbance; the training strategy is optimized, the disturbance is introduced in a scene and a stage, the reward signal is avoided due to the overlapping of events, and the causal relationship between the disturbance and the performance degradation can be accurately learned by the agent; the differentiated reward function is designed, the perception adjustment factor is introduced according to the attack type and severity, the targeted strategy optimization is realized, so that the robustness and adaptability of the controller in the diversified attack scene are improved. Specifically:

[0095] Firstly, a Markov decision process model is constructed for the secondary control problem of the microgrid, and the state space, action space and reward function of the system are defined; the state space includes key physical quantities reflecting the dynamic characteristics of the microgrid, such as system frequency, frequency deviation, key node voltage, voltage deviation and power of energy storage unit; the action space is defined as the control instruction output by the controller, specifically the active and reactive power output by the energy storage unit.

[0096] Secondly, a memory-enhanced reinforcement learning framework is proposed to improve the situation awareness ability and decision robustness of the control strategy; the core of the framework is to introduce a state adaptive input weighting mechanism, which is used to dynamically evaluate and quantify the importance of each state component in the state sequence; the mechanism calculates the short-term change amplitude of the state, generates an adaptive weight vector, and multiplies it with the original state sequence, so as to dynamically highlight the key event features such as voltage sag or frequency oscillation at the input level.

[0097] Then, the weighted state sequence is input into a long short-term memory network (LSTM); the LSTM network has excellent long sequence processing ability, which can deeply mine the long-term dependence relationship and potential mode of state data in the time dimension, so as to accurately identify the network attack features and operating condition mode; the hidden state output by the LSTM network is sent to the subsequent decision network as a deep representation of the environment situation.

[0098] Finally, a decision response is made in an actor-critic framework based on a double-delay deep deterministic policy gradient (TD3); the framework receives the enhanced situation awareness features extracted by the LSTM network and generates the optimal control action; in order to improve the learning efficiency and the robustness of the control strategy, the application designs a knowledge-assisted reward function, which not only punishes the deviation of frequency and voltage, but also introduces a dynamic penalty term related to the type and severity of network attacks; at the same time, a phased training strategy is adopted, first, the agent is trained under various operating conditions without attacks to establish a baseline performance, then different types of network attacks are introduced for special training in stages, and the exploration strategy is adjusted in stages during this process, so that the agent can accurately distinguish between external disturbances and malicious attacks, and gradually learn and master the stable control strategy in a complex coupled scenario.

[0099] The specific implementation process of the application includes:

[0100] The first step is to model the microgrid environment and define the Markov decision process (MDP). The microgrid model includes synchronous generators (SG), photovoltaic (PV) and battery energy storage systems (BESS) and other distributed power sources. The secondary control problem is described as an MDP, and the state space S is defined as a multi-dimensional vector containing the current system frequency , frequency deviation , previous time frequency deviation , point of common coupling (PCC) voltage , PCC voltage deviation , previous time PCC voltage deviation , node voltages of two energy storage units and and their voltage deviations and , and the active power inputs P1 and P2 of the two energy storage units; the action space A is defined as a four-dimensional control vector output by the controller, representing the active power and reactive power inputs for controlling the two energy storage units.

[0101] The second step is to build a memory-enhanced situation awareness module. The module consists of a state-adaptive input weighting mechanism and a long short-term memory (LSTM) network, and the core is to preprocess the input state sequence to highlight key information; specifically, for a state sequence of length T , where each state vector , is the dimension of the state space, and the state-adaptive input weighting mechanism calculates a corresponding weight vector for each state vector ; the process consists of the following sub-steps:

[0102] First, state change quantification is performed; to capture the dynamic changes of system states, the deviation of each state component from its recent history mean is calculated; specifically, for the i-th component in the state vector , its local mean is computed from the mean of the past time steps, with the formula:

[0103] .

[0104] Subsequently, the absolute difference between the current state component and the local mean is calculated, forming the deviation magnitude vector as:

[0105] .

[0106] This vector quantifies the fluctuation degree of each state dimension at the current time step.

[0107] Second, deviation feature transformation and smoothing are performed; to extract more representative features and reduce noise influence, the deviation magnitude vector is transformed by a learnable linear layer (defined by weight matrix and bias vector ) and a nonlinear activation function (e.g., ReLU function), resulting in the transformed feature vector ; then, a moving average filter with window size is applied to for smoothing, resulting in the smoothed deviation feature vector ; the calculation formula is as follows:

[0108] .

[0109] where: j represents the dimension index of the vector; this step converts the original fluctuation information into a more stable and robust feature representation.

[0110] Third, an adaptive amplification factor is generated; the smoothed deviation feature vector is multiplied by a sensitivity parameter and processed through a bounded activation function to generate the final adaptive amplification factor ; this bounded activation function limits the amplification effect within a preset maximum value to prevent excessive weight amplification; its formula is:

[0111] .

[0112] .

[0113] amplification factor The value of the amplification factor is positively correlated with the severity of the state change, that is, the greater the system state fluctuation, the greater the corresponding amplification factor value.

[0114] Finally, weights are selectively applied and a weighted state vector is generated; the amplification factor is not applied to all state components, but is selectively applied to state variables that are critical to system stability (such as frequency, PCC voltage, etc.) through a predefined binary mask vector The final adaptive weight vector is obtained by adding the reference weight (all-ones vector) to the masked amplification factor; the weighted state vector is obtained by element-wise multiplication of the original state vector and the weight vector ; the formula is as follows:

[0115] .

[0116] .

[0117] where: ⊙ denotes element-wise multiplication; through this series of steps, the mechanism can dynamically and selectively amplify key fluctuation information in the state sequence caused by network attacks or dramatic changes in operating conditions, thereby providing the subsequent LSTM network with more information-rich and higher signal-to-noise ratio inputs, significantly enhancing the system's incident perception capability.

[0118] Third, extract time series features through a long short-term memory network (LSTM). Input the weighted state sequence generated in the second step into an LSTM network to mine deep temporal dependencies between states; the reason why the LSTM network can effectively handle long sequence problems lies in its sophisticated gate unit structure, which includes a cell state and three gates, namely the forget gate, the input gate, and the output gate.

[0119] At each time step k in the sequence, the LSTM unit receives the current time's weighted state vector , the previous time's hidden state , and the previous time's cell state as input, and calculates the current time's hidden state and cell state ; the calculation process is as follows:

[0120] First, the forget gate determines which information needs to be discarded from the cell state at the previous time step ; for example, when the system has just experienced a large disturbance and is tending towards stability, the forget gate can help the network to reduce its focus on the previous, volatile state and focus on the current, stable mode of operation; the forget gate calculates an activation vector between 0 and 1 and , where each element of the vector corresponds to an information dimension in ; a value of "1" indicates complete retention, while a value of "0" indicates complete discarding; the formula for calculating the activation vector is as follows:

[0121] .

[0122] In the formula: , is a weight matrix, is a bias vector, and σ is a Sigmoid activation function.

[0123] Second, the input gate determines which new information will be stored in the current cell state; this enables the network to identify and focus on new key information, such as a sudden network attack signal or load switching event; this process involves two parts: the first part is the input gate layer, which determines which values need to be updated using a Sigmoid function; the second part is a tanh layer, which creates a new candidate value vector containing new information that can be added to the cell state; the formula for calculating the candidate value vector is as follows:

[0124] .

[0125] .

[0126] In the formula: , , , is a weight matrix, , is a bias vector.

[0127] Next, the cell state is updated; the old cell state is multiplied element by element with the output of the forget gate to discard the information that is decided to be discarded; then, the output of the input gate is multiplied element by element with the candidate value vector to obtain the information that needs to be added; finally, the two parts are added to obtain the new cell state at the current time step; the formula is as follows: ​

[0128] .

[0129] Finally, the Output Gate determines which part of the cell state will be output as the hidden state; the Output Gate first determines a filtering condition by a Sigmoid function ; then, the updated cell state is processed by a tanh function (scaling the value to -1 and 1) and multiplied element-wise with to obtain the final hidden state output at the current time step ; the formula is as follows:

[0130] .

[0131] .

[0132] Through the iterative loop at the time step, the LSTM network can encode and compress the time sequence information and key features of the entire input sequence into the hidden state at the last time step; this hidden state is a highly condensed feature vector that not only contains the current system state information, but also remembers the patterns and rules of historical state evolution, especially the key dynamics highlighted by the second weighting mechanism. Therefore, can provide a comprehensive and deep time insight system situation representation for the subsequent decision network, laying a solid foundation for the intelligent agent to make accurate and robust control decisions in a complex dynamic environment.

[0133] Fourthly, decision response and training optimization. The present application adopts an actor-critic architecture based on double-delay deep deterministic policy gradient (TD3), and combines knowledge-assisted reward function design and phased training strategy to realize efficient and robust control decisions.

[0134] Firstly, at the network structure level, the Actor network and the Critic network both receive the hidden state sequence generated in the third step as input, and output decisions after nonlinear processing by additional fully connected layers; the Actor network outputs a deterministic action according to the current policy function μ, and adds Gaussian noise in the training process to promote policy exploration; the action generation formula is:

[0135] .

[0136] In the formula: The parameters of the Actor network.

[0137] Secondly, in order to effectively guide the agent learning, a knowledge-aided compound reward function is designed ; The function takes into account multiple control objectives, and its expression is:

[0138] .

[0139] Where each component is defined as follows:

[0140] : Punish the voltage deviation of the public coupling point (PCC).

[0141] : Punish the voltage deviation of the two energy storage unit nodes.

[0142] : Punish the system frequency deviation.

[0143] : When the agent's action causes the system to enter extreme operating conditions, a binary penalty is applied.

[0144] : Apply a small penalty proportional to the simulation time step to encourage the agent to take more efficient control actions.

[0145] : This is a network attack-aware penalty term, with an attack scaling factor According to the type of attack, it is dynamically adjusted, and when there is no attack, it is 0, and under random signal attack (RSA), decaying signal attack (DSA) and periodic signal attack (PSA), it takes different positive values, so as to pay more attention to the stability of the system when attacked.

[0146] Thirdly, during the training update, the agent stores the experience tuples generated during the interaction process in the experience replay pool; a small batch of data is randomly sampled from it during training; the TD3 algorithm uses a pair of Critic networks ( , ) and their corresponding target networks ( , ) to suppress the overestimation of Q values; the calculation of the target Q value uses the Clipped Double-Q Learning technology, which is expressed as:

[0147] .

[0148] In the formula, is the discount factor; the action is the next time action generated by the target Actor network with added clipping noise; each Critic network is updated by minimizing the mean squared Bellman error, whose loss function is:

[0149] .

[0150] The Actor network is updated by policy gradient, which uses the output of the first Critic network to guide the improvement direction of the policy, and its policy gradient is approximated as:

[0151] .

[0152] The parameters of all target networks (target Actor network and two target Critic networks) are slowly approaching the current network parameters θ through a soft update mechanism to ensure the stability of training, and its update formula is:

[0153] .

[0154] where τ is a soft update coefficient much smaller than 1.

[0155] Finally, a phased training strategy is adopted; in the initial stage, the agent is trained under various operating conditions without any network attack to learn the basic frequency and voltage control capability; after completing this stage, it enters the special training stage for specific attack types in turn; this training sequence from easy to difficult, combined with the attack perception reward function, enables the agent to gradually distinguish between regular disturbances and malicious attacks; at the same time, when entering the attack special training stage, the strength of the exploration noise will be appropriately reduced, so as to stabilize the learning process in the adversarial environment; through this series of refined design, an agent that can achieve efficient and robust secondary control in a complex, dynamic and network security threatened microgrid environment is ultimately obtained.

[0156] In the specific simulation experiment, as Figure 2 shows the evolution of rewards in the phased reinforcement learning training process; in the first stage, the agent learns under various operating conditions, and the reward increases rapidly from about −600 to about −60, indicating that the baseline control strategy is effectively established; then, in the 2a stage, when facing RSA attacks, the reward initially decreases and then gradually recovers to about −50; the 2b stage shows the adaptation process to DSA attacks, and the reward increases from −300 to −50; the final 2c stage reveals significant fluctuations when facing PSA attacks, and the reward first drops sharply to about −400 and then quickly recovers to above −50, indicating that the agent has successfully adapted to all attack types and developed a robust defense strategy; the overall trend verifies the effectiveness of the phased training strategy and the proposed memory-enhanced reinforcement learning controller.​

[0157] The circuit breaker disconnects after 30 seconds and then reconnects after 80 seconds; the photovoltaic generator initially outputs 0.5 pu when the system starts up, and its output increases to 1 p.u. after 50 seconds.

[0158] Figure 3 The system response under RSA in the case study is shown. The controller proposed in this invention maintains frequency and voltage stability with minimal deviation, achieving a standard deviation of 0.00660 pu and a yield rate of 99.56%. In contrast, the PID and DDPG controllers exhibit significant instability, especially after feeder reconnection at t=80s. Figure 4 The comparison results under PSA are shown. The method of the present invention achieves stable performance (excellent rate of 99.57%), which is superior to other control strategies. The controller of the present invention quickly restores the frequency and voltage to the rated level after each disturbance.

[0159] This invention significantly enhances the system's situational awareness through a state-adaptive input weighting mechanism, dynamically highlighting key state changes and effectively capturing critical events such as rapid voltage drops or frequency oscillations occurring under network attacks or operational failures. Combined with the deep mining capabilities of LSTM networks for long-term dependencies, this technology can accurately identify attack patterns and operating conditions, significantly improving the intelligent sensing level of the microgrid. In terms of control performance, this invention demonstrates superior performance, significantly reducing system frequency and voltage fluctuations under various network attack scenarios. Compared to traditional methods, it achieves optimal performance in key indicators such as standard deviation, average deviation, and success rate. For example, under RSA and PSA attacks, the success rate is over 99.5% (deviation within 0.03 pu is considered excellent), and the standard deviation is only 0.00660 pu. Furthermore, thanks to the phased training strategy, the system not only converges quickly and rapidly recovers stability under various attacks and complex operating conditions, but its single-step inference time is also only 0.0134 seconds, with moderate computational complexity, fully meeting the requirements of real-time control.

[0160] This invention can simultaneously cope with multi-mode and even unknown network attacks and dynamically changing operating conditions without adjusting parameters for specific systems, and has good scalability. Through knowledge-assisted reward design and phased training, the robustness of the control strategy is significantly enhanced, especially when dealing with complex long sequence correlation problems. Compared with other recurrent neural networks, the LSTM architecture converges faster and has more stable performance when dealing with complex sequences, and can achieve comprehensive coordinated control of frequency and voltage, effectively addressing the coupling threats from the network and physical layers.

[0161] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A secondary control method for a new energy microgrid, characterized in that, Includes the following steps: The secondary control process of the new energy microgrid is constructed as a Markov decision process, wherein the state space includes a state vector composed of physical quantities characterizing the dynamic characteristics of the new energy microgrid, and the action space includes control commands output by the controller that characterizes the secondary control of the new energy microgrid. Historical state sequences at multiple time steps are collected during the operation of the new energy microgrid. For each state vector in the state sequence, the mean of multiple state variables in the state vector is obtained, the deviation between the state variables and the mean of the state variables is calculated, and a deviation amplitude vector is formed based on multiple deviations. The deviation amplitude vector is then transformed through a linear transformation layer defined by a weight matrix and a bias vector and a nonlinear activation function to obtain a deviation feature vector that characterizes the importance of the state variables in the state vector. The adaptive amplification factor is obtained by multiplying the deviation feature vector with a preset sensitivity parameter; the adaptive amplification factor is selectively applied to the state variables in the state vector that are associated with the stability of the microgrid through a predefined binary mask vector to obtain the adaptive weight vector; the state vector and the adaptive weight vector are multiplied element by element to obtain the weighted state vector and the weighted state vector sequence. The weighted state vector sequence is passed through a Long Short-Term Memory (LSTM) network to capture the deep temporal dependencies of the state data in the state vector sequence and obtain the hidden state of the weighted state vector sequence. Based on the hidden state, control decision responses are made to determine the controller's control commands for secondary control of the new energy microgrid.

2. The secondary control method for a new energy microgrid according to claim 1, characterized in that, The acquisition of the deviation feature vector includes: Collect a state sequence of length T during the operation of a new energy microgrid. , where each state vector , The dimension of the state space; For the state vector The i-th component The mean of its state components From the past The average value of each time step is calculated using the following formula: ； Calculate the absolute difference between the current state component and the mean of the state components to form the deviation magnitude vector. for: ； Deviation amplitude vector Through a learnable linear layer and a nonlinear activation function Perform the transformation to obtain the transformed feature vector. ; and on Apply a window with a size of The moving average filter is used for smoothing to obtain the smoothed deviation eigenvector. The formula is as follows: ； Where: j represents the dimension index of the vector.

3. The secondary control method for a new energy microgrid according to claim 2, characterized in that, The obtained weighted state vector includes: The smoothed deviation eigenvector Multiply by a sensitivity parameter and through a bounded activation function The process is performed to generate the final adaptive scaling factor. A bounded activation function limits the amplification effect to a preset maximum value. Within this range, the formula is: ； ； Through a predefined binary mask vector Selectively applied to state variables associated with control; Adaptive weight vector The original state vector is obtained by adding the baseline weights and the magnification factor after masking. With adaptive weight vector Multiplying element by element yields the weighted state vector. The formula is expressed as: ； ； Where: ⊙ represents element-wise multiplication.

4. The secondary control method for a new energy microgrid according to claim 3, characterized in that, The hidden states of the obtained weighted state vector sequence include: The weighted state vector sequence is It is then fed into a Long Short-Term Memory (LSTM) network; In the state vector sequence At each time step k, the forget gate will be based on the weighted state vector at the current time. and the hidden state of the previous moment Calculate an activation value vector between 0 and 1. Each element of this vector corresponds to the cell state at the previous time step. One of the information dimensions; "1" indicates complete retention, while "0" indicates complete rejection; its calculation formula is expressed as: ； in: and Represents the weight matrix; σ represents the bias vector; σ represents the Sigmoid activation function. The input gate uses the sigmoid function to determine which values ​​need to be updated, and a new candidate value vector is created through a tanh layer. This vector contains new information that may be added to the cell state; the calculation formula is as follows: ； ； in: , , and Represents the weight matrix; and Represents the bias vector; old cell state Output of the forget gate Element-wise multiplication, discarding the information to be discarded; the output of the input gate... With candidate value vector Multiply each element to obtain the information that needs to be added; add these two parts together to obtain the new cell state at the current moment. The formula is expressed as: ； The output gate uses the Sigmoid function to determine a filtering condition. Updated cell state Processed using the tanh function, and then... Multiply each element to obtain the final hidden state output at the current moment. The formula is expressed as: ； 。 5. The secondary control method for a new energy microgrid according to claim 4, characterized in that, When making control decision responses based on hidden states, an Actor-Critic architecture based on the dual-delay deep deterministic policy gradient TD3 is adopted. Based on the initial reward function of the Markov decision process, a knowledge-assisted reward function is constructed to perform control decision responses, including: Both the Actor network and the Critic network receive the hidden state sequence. As input, the decision is output after nonlinear processing by a fully connected layer; The Actor network outputs a deterministic action based on the current policy function μ. Gaussian noise was added during the training process. To facilitate strategy exploration; the action generation formula is: ； in: These are the parameters of the Actor network; Set up a knowledge-assisted composite reward function ,function Taking into account multiple control objectives, its expression is: ； Each component is defined as follows: : Penalize voltage deviation at the point of common coupling (PCC); : Penalize the voltage deviation between the two energy storage unit nodes; : Penalize system frequency deviation; When an agent's actions cause the system to enter extreme operating conditions, a binary penalty is imposed. Apply a small penalty proportional to the simulation time step to encourage the agent to take more efficient control actions; This is a network attack awareness penalty item, with its attack scaling factor. The value is dynamically adjusted based on the attack type. It is 0 when there is no attack, and takes different positive values ​​under random signal attack (RSA), attenuated signal attack (DSA), and periodic signal attack (PSA), thereby increasing the focus on system stability when an attack occurs.

6. The secondary control method for a new energy microgrid according to claim 5, characterized in that, This also includes training the constructed Markov decision process, including: During training, the agent will use the experiential tuples generated during the interaction process. Store the data in the experience replay pool, and randomly sample small batches of data from it during training. The TD3 algorithm utilizes a pair of Critic networks ( , ) and its corresponding target network ( , To suppress overestimation of the Q value; target Q value The calculation uses the Clipped Double-Q Learning method, expressed as: ； in: Indicates discount factor; action It is the next-moment action generated by the target Actor network with pruned noise added; Each Critic network is updated by minimizing the mean square Bellman error, and its loss function is: ； The Actor network is updated using the policy gradient, and the output of the first Critic network guides the direction of policy improvement. Its policy gradient is approximately: ； Parameters of all target networks All parameters gradually approach the current network parameter θ through a soft update mechanism, and the update formula is as follows: ； Where: τ is a soft update coefficient much smaller than 1; A phased training strategy is adopted: in the initial stage, the agent is trained under various operating conditions without any network attacks. After this stage is completed, it enters a special training stage for specific attack types. When entering the attack-specific training stage, the intensity of exploration noise is reduced to stabilize the learning process and complete the training.

7. The secondary control method for a new energy microgrid according to claim 1, characterized in that, The construction of the Markov decision process includes: An islanded microgrid model is constructed, which includes a synchronous generator (SG), a photovoltaic (PV) system, and a battery energy storage system (BESS). The secondary control problem of the microgrid model is modeled as a Markov decision process. The state space S is defined as containing the current system frequency. Frequency deviation Frequency deviation at the previous moment Common point of coupling (PCC) voltage PCC voltage deviation PCC voltage deviation at the previous moment The node voltages of the two energy storage units and and its voltage deviation and , and the multidimensional vectors of the active power inputs P1 and P2 of the two energy storage units; Action space A is defined as the four-dimensional control vector output by the controller. , representing the active power and reactive power inputs for controlling the two energy storage units, respectively.

8. A secondary control device for a new energy microgrid, characterized in that, include: The agent module is used to construct the secondary control process of the new energy microgrid as a Markov decision process. The state space includes a state vector composed of physical quantities that characterize the dynamic characteristics of the new energy microgrid, and the action space includes control commands output by the controller that characterize the secondary control of the new energy microgrid. The situational awareness module is used to collect historical state sequences at multiple time steps during the operation of the new energy microgrid. For each state vector in the state sequence, the mean of multiple state variables in the state vector is obtained, the deviation between the state variables and the mean of the state variables is calculated, and a deviation magnitude vector is formed based on multiple deviations. The deviation magnitude vector is then transformed through a linear transformation layer defined by a weight matrix and a bias vector and a nonlinear activation function to obtain a deviation feature vector that characterizes the importance of the state variables in the state vector. The adaptive amplification factor is obtained by multiplying the deviation feature vector with a preset sensitivity parameter; the adaptive amplification factor is selectively applied to the state variables in the state vector that are associated with the stability of the microgrid through a predefined binary mask vector to obtain the adaptive weight vector; the state vector and the adaptive weight vector are multiplied element by element to obtain the weighted state vector and the weighted state vector sequence. The weighted state vector sequence is passed through a Long Short-Term Memory (LSTM) network to capture the deep temporal dependencies of the state data in the state vector sequence and obtain the hidden state of the weighted state vector sequence. The decision control module makes control decisions based on the hidden state and determines the controller's control commands to perform secondary control of the new energy microgrid.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the steps of the secondary control method for a new energy microgrid as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the steps of a secondary control method for a new energy microgrid as described in any one of claims 1 to 7.

Images