Intelligent power grid intrusion security tracking control method and system based on reinforcement learning

Abstract

The application belongs to the technical field of safety control of micro-grid systems, and specifically discloses an intelligent power grid containment safety tracking control method and system based on reinforcement learning. In order to reduce the influence caused by DoS attacks, the last successful transmission state information is used as a compensation item in the controller and the event triggering mechanism. The event triggering scheme proposed in the application not only can save communication resources, but also can effectively reduce the risk of being attacked by DoS, and the event trigger can work normally without Zeno behavior. Through the compensation mechanism and the model-free algorithm based on reinforcement learning, the application solves the problem of completely distributed event voltage tracking control of the micro-grid system under DoS attacks.

Description

Technical Field

[0001] This invention belongs to the field of security control technology for microgrid systems, and specifically relates to a smart grid intrusion-tolerant security tracking control method and system based on reinforcement learning. Background Technology

[0002] Microgrid systems are a crucial component connecting distributed generators (DGs) and the power system, and as a type of multi-agent system (MAS), they have become a hot research topic. As a building unit, a microgrid is a small-scale power system that facilitates the significant integration of distributed grids. With proper control, microgrids can operate stably and economically. Centralized control, the earliest framework to achieve secondary control, utilizes all information from each DG, but this approach may lead to instability in the microgrid system.

[0003] To address the consistency problem, fully distributed control has been proposed in recent years. Unlike traditional distributed control, fully distributed control can eliminate the requirement for a minimum non-zero eigenvalue that is relevant to the entire network topology.

[0004] Denial-of-service (DoS) attackers clog communication channels by sending large numbers of useless data packets, preventing the transmission of correct information and thus destabilizing or even destroying the entire microgrid system.

[0005] The reliability and security of microgrid systems are crucial for the stable operation of critical infrastructure such as energy supply and transportation. Within a microgrid system, various devices and components are networked together, including controllers and sensors. Therefore, a DoS attack could render these devices unable to communicate or function, causing the entire microgrid system to fail.

[0006] In addition, while continuous communication enables the microgrid system to operate stably, it also brings problems such as large computational load and waste of resources. Summary of the Invention

[0007] The purpose of this invention is to propose a smart grid intrusion-tolerant security tracking control method based on reinforcement learning. This method, by employing a compensation mechanism and a model-free algorithm based on reinforcement learning, is beneficial for solving the fully distributed event voltage tracking control problem of microgrid systems under DoS attacks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A smart grid intrusion-tolerant safety tracking and control method based on reinforcement learning includes the following steps:

[0010] Step 1. Establish a dynamic model for generator secondary voltage tracking control in a microgrid system;

[0011] Step 2. Establish a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and how they affect smart grid systems. Based on the mechanism of DoS attacks and how they affect systems, provide piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy.

[0012] Step 3. Design an event trigger to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller designed in Step 5 below;

[0013] Step 4. Calculate the control gain of the reinforcement learning-based controller by selecting positive constants;

[0014] Step 5. Based on steps 1 to 4, design an event-triggered fully distributed controller with an attack compensation mechanism to resist the impact of network attacks on the system while conserving network communication resources.

[0015] Step 6. Based on the control gain obtained in Step 4 and the attack compensation mechanism designed in Step 5, and combined with the event trigger designed in Step 3, an event-triggered elastic controller is obtained to realize the voltage tracking control task.

[0016] Furthermore, based on the reinforcement learning-based smart grid intrusion tolerance safety tracking and control method, this invention also proposes a corresponding reinforcement learning-based smart grid intrusion tolerance safety tracking and control system, which adopts the following technical solution:

[0017] A smart grid intrusion tolerance safety tracking and control system based on reinforcement learning includes:

[0018] The dynamic model building module is used to establish a dynamic model for generator secondary voltage tracking control in a microgrid system.

[0019] The attack model construction module establishes a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and their impact on smart grid systems. Based on the mechanism of DoS attacks and their impact, it provides piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy.

[0020] The event trigger building module designs event triggers to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller designed below.

[0021] The control gain module calculates the control gain of the reinforcement learning-based controller by selecting a positive constant.

[0022] The controller building module is designed with an attack compensation mechanism to resist the impact of network attacks on the system while conserving network communication resources.

[0023] The voltage tracking control module, based on the control gain and the designed attack compensation mechanism, combined with the designed event trigger, obtains an event-triggered elastic controller to realize the voltage tracking control task.

[0024] Furthermore, based on the aforementioned reinforcement learning-based smart grid intrusion-tolerant security tracking and control method, this invention also proposes a computer device, which includes a memory and one or more processors.

[0025] The memory stores executable code, and when the processor executes the executable code, it implements the steps of the reinforcement learning-based smart grid intrusion-tolerant security tracking and control method described above.

[0026] Furthermore, based on the aforementioned reinforcement learning-based smart grid intrusion-tolerant security tracking and control method, this invention also proposes a computer-readable storage medium storing a program thereon. When executed by a processor, this program is used to implement the steps of the reinforcement learning-based smart grid intrusion-tolerant security tracking and control method described above.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] As described above, this invention discloses a reinforcement learning-based smart grid intrusion-tolerant security tracking control method. This invention overcomes the need for global topology information by utilizing a proposed fully distributed strategy. This invention proposes a model-free algorithm based on reinforcement learning. To mitigate the impact of DoS attacks, this invention uses the last successfully transmitted state information as a compensation term in both the controller and the event triggering mechanism. The event triggering scheme proposed in this invention not only saves communication resources but also effectively reduces the risk of DoS attacks, and the event trigger can function normally without Zeno behavior. This invention, by employing a compensation mechanism and a model-free algorithm based on reinforcement learning, effectively solves the problem of fully distributed event voltage tracking control in microgrid systems under DoS attacks. Attached Figure Description

[0029] Figure 1 This is a flowchart of a smart grid intrusion-tolerant security tracking and control method based on reinforcement learning in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the attack mechanism in an embodiment of the present invention. Detailed Implementation

[0031] Example 1

[0032] This embodiment 1 describes a smart grid intrusion-tolerant security tracking control method based on reinforcement learning, which addresses the fully distributed event-triggered secondary voltage tracking control problem of a microgrid system subjected to a DoS attack.

[0033] like Figure 1 As shown, the smart grid intrusion tolerance safety tracking control method based on reinforcement learning includes the following steps:

[0034] Step 1. Establish a dynamic model for generator secondary voltage tracking control in a microgrid system.

[0035] The dynamic model for generator secondary voltage tracking control in a microgrid system is as follows:

[0036]

[0037] Where i represents the i-th generator, and i is a natural number greater than 0; z i (t) and ζ i (t) represents the state vector and control input of the i-th generator in the microgrid system, respectively; A and B are the system matrix and input matrix, respectively.

[0038] Meanwhile, the reference signal takes the following form:

[0039] in, And ξ0=v ref Let v be a time-invariant constant. ref This indicates the reference voltage value.

[0040] Step 2. Establish a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and their impact on smart grid systems. Based on the mechanism of DoS attacks and their impact, provide piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy.

[0041] This invention divides the entire time interval into two parts to analyze the impact of DoS attacks on smart grid systems.

[0042] like Figure 2 As shown, the entire time interval is divided into a safe interval s and an attacked interval a. For these two intervals, the following piecewise function is designed to handle the system in these two situations.

[0043] For a time interval starting at time t0, the malicious network attack model can be represented as the following piecewise function.

[0044] Indicates the entire time interval during which the attack occurred; Represents the union, The set of positive integers; ∩ represents the intersection; Π represents the time interval in which the (m+1)th attack occurs; s (t0,t)=[t0,t) / Π a (t0,t) represents the safe time interval.

[0045] t0 is the initial time and That is, it is assumed that the initial time is within a safe time interval.

[0046] Given that attackers have limited energy, the duration of a network attack within a finite time interval (t0,t) is called the attack duration Π. a Both (t0,t) and the attack frequency Φ(t0,t) are subject to constraints:

[0047]

[0048]

[0049] Where, N a τ a N b and τ b All are normal numbers.

[0050] Step 3. Design an event trigger to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller designed in Step 5 below.

[0051] The design of an event-triggered fully distributed controller with an attack compensation mechanism is as follows:

[0052] For the i-th DG, the triggering error is given as follows:

[0053]

[0054] Where, σ i (t) represents the triggering error of the i-th DG, t i m This represents the m-th trigger time of the i-th agent. This represents the voltage state value sent by the event trigger for the mth time.

[0055] The attack compensation mechanism for dealing with DoS attacks is given below, as shown in the following formula;

[0056]

[0057] in, The trigger time of DG j This is the moment of the last successful data transmission before a DoS attack occurs. The sampled value of the event trigger of DG j, This is the state value of the last transmission before the attack occurred.

[0058] Based on this, the following fully distributed event-triggered controller with an attack compensation mechanism is proposed:

[0059]

[0060]

[0061] in,

[0062] N represents the number of generators, a ij This indicates the communication status between DGi and DGj; if communication exists between DGi and DGj, then a ij =1, otherwise, a ij =0. K represents the control gain, which is obtained through the subsequent reinforcement learning algorithm, and has:

[0063] Θ=K T K.

[0064] n ij (t) represents the time-varying coupling weights, and we have:

[0065] n ij (0)=n ji (0)>0,n ij (t)=n ji (t);

[0066]

[0067]

[0068] Where, p ij q ij Let p be the coupling parameters, all of which are positive constants. ij =p ji ,q ij =q ji The event triggering function for each DG is given below, as shown in the following formula:

[0069]

[0070] Among them, f i (t) represents the event triggering function of the i-th DG.

[0071]

[0072]

[0073] Wherein, parameter w s α s β s w a α a and β a All are positive integers, n i0 (t) represents the leader coupling weight value of agent i, a i0 This represents the communication state between agent i and the leader.

[0074] Parameter w s w a α s α a β a and β s All are time-invariant system parameters that are greater than zero.

[0075] The event triggering function designed in this invention is defined as an error-related function, and its function value increases with the error.

[0076] When the growth exceeds 0, the event trigger in step 3 will perform a triggering action to send the relevant data of the distributed generator to other generators and use it to update the fully distributed controller in step 5.

[0077] Define the event triggering time sequence as And there are for The initial value of .

[0078] In this event trigger time series, the condition for updating the next trigger time is defined as:

[0079] 1) The time exceeds the previous trigger time;

[0080] 2) The event triggering function is greater than zero.

[0081] Under this update rule, each agent's event trigger detector monitors the trigger function value in real time. When the trigger function value is greater than zero, a trigger is performed, broadcasting the state value of this agent to all other agents, thereby reducing the use of network resources during communication.

[0082] This invention addresses the problems caused by uninterrupted communication by proposing an event-triggered control method.

[0083] Traditional control methods typically involve periodically checking the system state to execute corresponding control operations. This method wastes a lot of resources, especially when the system state remains unchanged.

[0084] The event-triggered mechanism can trigger corresponding control operations when the system state changes, and has the following advantages:

[0085] Improved control precision: Event-triggered mechanisms enable the controller to respond immediately when the system state changes, thereby controlling the system more accurately.

[0086] Improving system robustness: In microgrids, the system state may fluctuate due to various factors, such as load changes and motor failures. By introducing an event-triggered mechanism, the controller can respond to these changes in a timely manner, thereby improving the system's robustness and enabling it to better cope with various abnormal situations.

[0087] Enhancing system flexibility: Event-triggered mechanisms allow controllers to respond more flexibly to changes in system state, enabling them to adopt different control strategies for different situations, thereby further improving the system's adaptability and flexibility.

[0088] In summary, the method of the present invention, by introducing an event triggering mechanism, can make microgrid control technology more intelligent, efficient, and flexible, thereby providing a more reliable guarantee for the stable operation of microgrids.

[0089] Step 4. By selecting a positive constant, the control gain of the reinforcement learning-based controller is calculated. Applying this control gain can ultimately achieve the voltage safety tracking control objective.

[0090] First, define the control input for each DG as υ. i (t)=Kz i (t), we have:

[0091] And υ i (t)=-ζ i (t).

[0092] Among them, υ i (t) represents the control input of the i-th DG, and K represents the control gain.

[0093] The design process of a controller based on reinforcement learning is as follows:

[0094] First, choose any initial state z. i , and selected And P = I.

[0095] in, express The initial values ​​are given by P, where P represents the positive definite parameter matrix and I represents the identity matrix.

[0096] Obtained using system input and output data Use Update Eventually, the conditions will be met. Data written to In, or output that meets the conditions.

[0097] in, Represents the parameter matrix The value of the (k+1)th iteration. Represents the parameter matrix Γ i The value of the kth iteration. Represents the parameter matrix K i The value of the kth iteration, ρ k Indicates the parameter value.

[0098] Represents the parameter matrix S i The value of the (k+1)th iteration; Represents the parameter matrix S i The value of the kth iteration.

[0099] First, determine Does it belong to the predefined bounded set C? m .

[0100] If it does not belong, then Reset to initial value If it belongs to the category, then further determine its relationship with... Does the norm of the difference satisfy the requirements? If satisfied, output As the control gain K.

[0101] If the condition is not met, then the condition that is met will be... Data written to In this context, δ represents a predefined normal parameter.

[0102] Step 5. Based on steps 1 to 4, design an event-triggered fully distributed controller with an attack compensation mechanism to resist the impact of network attacks on the system while saving network communication resources, and finally achieve voltage safety tracking control. Its fully distributed characteristic is reflected in avoiding the use of global topology information.

[0103] The controller input u of the event-triggered elastic controller i (t) is represented as:

[0104]

[0105] in, A flag indicating that an attack has occurred.

[0106]

[0107] This represents the k-th trigger time of distributed generator i. This represents the k'-th trigger time of distributed generator j. This is the moment of the last successful data transmission before a DoS attack occurs.

[0108] Step 6. Based on the control gain obtained in Step 4 and the attack compensation mechanism designed in Step 5, and combined with the event trigger designed in Step 3, an event-triggered elastic controller is obtained to realize the voltage tracking control task.

[0109] Specifically, the process of achieving voltage tracking control is as follows:

[0110] By comparing the voltage status information of its own generator in the event-triggered elastic controller with the voltage status information received from neighboring generators and the reference voltage information, the voltage status information of its own generator is continuously adjusted to reduce the difference between its own generator voltage and the reference voltage, thus completing the voltage tracking control task.

[0111] like Figure 1 A control flowchart of the method of the present invention is shown. Wherein, the voltage state value z of distributed generator i... i (t) is collected by the sensor, sampled by the event trigger, and sent to the transmitter. The transmitter is responsible for broadcasting the received data to the receivers of other intelligent agents, including itself. An attacker can target this action. When an attack occurs, the transmitter's function is disabled, and the intelligent agent will not be able to receive the voltage state values ​​of neighboring intelligent agents. The receiver can then send the received information to the controller, which generates a control signal u. i (t) is sent to the actuator to adjust the voltage value of the distributed generator.

[0112] Compared with traditional microgrid secondary voltage elastic tracking control schemes, the present invention has the following advantages:

[0113] 1) Previous distributed strategies for DoS attacks require global information related to network communication topology. This invention proposes a novel fully distributed DoS attack control method. Based on this, a fully distributed resilient control strategy is proposed to complete the voltage tracking task. Furthermore, the event-triggered mechanism reduces the waste of communication resources and the negative impact of DoS attacks. 2) In reality, due to the complexity of the system, system information is not always available. The RL reinforcement learning algorithm only uses the initial values ​​of the system state to ensure the algebraic Riccati equation holds. In addition, the proposed attack compensation mechanism mitigates the negative impact of DoS attacks, thereby maintaining system stability. The latest successfully transmitted state information of neighbors is used as an attack compensation term, and a reinforcement learning-based attack resilient controller is designed.

[0114] This invention utilizes a proposed fully distributed strategy that avoids the use of global topology information. First, based on the DoS model described in step 2, this invention uses the state information of the last successfully transmitted data as a compensation term to mitigate the impact of DoS attacks. The event trigger in step 3 can function normally without Zeno behavior, thus conserving network communication resources. Furthermore, in step 4, this invention proposes a model-free algorithm based on reinforcement learning. By selecting positive constants, the control gain of the reinforcement learning-based controller is calculated, and this control gain ultimately achieves the voltage tracking control objective. Finally, based on the control gain obtained in step 4 and the attack compensation mechanism designed in step 5, combined with the event trigger designed in step 3, an event-triggered elastic controller is obtained. By comparing the voltage state information of its own generator with the received voltage state information from neighboring generators and the reference voltage information, the event-triggered elastic controller continuously adjusts its own generator's voltage state information to reduce the difference between its own generator's voltage and the reference voltage, ultimately completing the voltage tracking control task.

[0115] Example 2

[0116] This embodiment 2 describes a smart grid intrusion tolerance safety tracking and control system based on reinforcement learning. This system is based on the same inventive concept as the smart grid intrusion tolerance safety tracking and control method based on reinforcement learning in embodiment 1 above.

[0117] Specifically, the reinforcement learning-based smart grid intrusion tolerance security tracking and control system includes:

[0118] The dynamic model building module is used to establish a dynamic model for generator secondary voltage tracking control in a microgrid system.

[0119] The attack model construction module establishes a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and their impact on smart grid systems. Based on the mechanism of DoS attacks and their impact, it provides piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy.

[0120] The event trigger building module designs event triggers to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller designed below.

[0121] The control gain module calculates the control gain of the reinforcement learning-based controller by selecting a positive constant.

[0122] The controller building module is designed with an attack compensation mechanism to resist the impact of network attacks on the system while conserving network communication resources.

[0123] The voltage tracking control module, based on the control gain and the designed attack compensation mechanism, combined with the designed event trigger, obtains an event-triggered elastic controller to realize the voltage tracking control task.

[0124] It should be noted that the implementation process of the functions and roles of each functional module in the reinforcement learning-based smart grid intrusion-tolerant security tracking and control system is detailed in the implementation process of the corresponding steps in the method of the above embodiment 1, and will not be repeated here.

[0125] Example 3

[0126] This embodiment 3 describes a computer device used to implement the steps of the reinforcement learning-based smart grid intrusion-tolerant security tracking and control method described in embodiment 1 above.

[0127] The computer device includes a memory and one or more processors. Executable code is stored in the memory, and when the processor executes the executable code, it implements the steps of a reinforcement learning-based smart grid intrusion-tolerant security tracking control method.

[0128] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0129] Example 4

[0130] This embodiment 4 describes a computer-readable storage medium for implementing the steps of the reinforcement learning-based smart grid intrusion-tolerant security tracking and control method described in embodiment 1 above.

[0131] The computer-readable storage medium in this embodiment 4 stores a program that, when executed by a processor, implements the steps of a reinforcement learning-based smart grid intrusion-tolerant security tracking and control method.

[0132] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0133] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A smart grid intrusion-tolerant safety tracking control method based on reinforcement learning, characterized in that, Includes the following steps: Step 1. Establish a dynamic model for generator secondary voltage tracking control in a microgrid system; The dynamic model for generator secondary voltage tracking control in a microgrid system is as follows: ； Where i represents the i-th generator, and i is a natural number greater than 0; and Let these represent the state vector and control input of the i-th generator in the microgrid system, respectively. and These are the system matrix and the input matrix, respectively. The reference signal is in the form of: ; in, And there are Let be a time-invariant constant. Indicates the reference voltage value; Step 2. Establish a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and how they affect smart grid systems; based on the mechanism of DoS attacks and how they affect systems, give piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy. The time interval is divided into a safe interval s and an attacked interval a to analyze the impact of DoS attacks on smart grid systems; the following piecewise function is designed to handle the system in these two cases. For The malicious network attack model is represented by the following piecewise function within a time interval starting from the initial time. : ； Indicates the entire time interval during which the attack occurred; Represents the union, set of positive integers; Indicates intersection; Indicates the first The time interval during which the attack occurred; Indicates a safe time interval; For the initial time and That is, it is assumed that the initial time belongs to the safe time interval; Given the fact that attackers have limited energy, within a finite time interval The duration of a cyberattack, also known as the attack duration. and attack frequency All have constraints: ； ； in, , , and All are positive numbers; Step 3. Design an event trigger to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller in Step 5 below; Step 4. Calculate the control gain of the reinforcement learning-based controller by selecting positive constants; Step 5. Based on steps 1 to 4, design an event-triggered fully distributed controller with an attack compensation mechanism to resist the impact of network attacks on the system while conserving network communication resources. The design of an event-triggered fully distributed controller with an attack compensation mechanism is as follows: First, for the i-th generator, the triggering error is given as follows: ； in, This represents the triggering error of the i-th generator. This represents the m-th trigger time of the i-th agent. This represents the voltage state value sent by the event trigger for the mth time. The attack compensation mechanism for dealing with DoS attacks is given below, as shown in the following formula; ； in, Let j be the trigger time of the j-th generator; This is the moment of the last successful data transmission before a DoS attack occurs. The sampled value of the event trigger for the j-th generator. The state value transmitted last before the attack occurred; based on this, the following fully distributed event-triggered controller with an attack compensation mechanism is proposed: ； ； in, , ; N represents the number of generators. Indicates the first The generator and the first The communication status of the generator; if the first generator... The generator and the first If there is communication between the generators, then ,otherwise, ; K represents the control gain, which is obtained through a reinforcement learning algorithm, and has... ; Let be the time-varying coupling weights, and we have: , ； ； ； in, , These are coupling parameters, all of which are positive constants, and have... , ; The event triggering function for each generator is given below, as shown in the following formula: ； in, This represents the event trigger function for the i-th generator; ； ； Among them, parameters , , , , as well as All are positive numbers. Represents intelligent agents Leader coupling weight value, Represents intelligent agents Communication status with leaders; The event trigger function is defined as an error-related function. Its function value increases with the error. When the increase exceeds 0, the event trigger in step 3 will perform a triggering action to send the relevant data of the distributed generator to other generators and use it to update the fully distributed controller in step 5. Define the event triggering time sequence as And there are , for The initial value; In this event triggering time series, the condition for updating the next trigger time is defined as: 1) The time has exceeded the previous trigger time; 2) The event trigger function is greater than zero; Under this update rule, each agent's event trigger detector detects the trigger function value in real time. When the trigger function value is greater than zero, a trigger is performed, and the state value of this agent is broadcast to all other agents. Step 6. Based on the control gain obtained in Step 4 and the attack compensation mechanism designed in Step 5, and combined with the event trigger designed in Step 3, an event-triggered elastic controller is obtained to realize the voltage tracking control task.

2. The smart grid intrusion tolerance safety tracking and control method according to claim 1, characterized in that, Step 4 specifically involves: First, define the control input for each generator as... ,have: and ; in, This represents the control input of the i-th generator. Indicates control gain; The design process of a controller based on reinforcement learning is as follows: First, choose any initial state. , and selected and ; in, express initial value, Denotes a positive definite parameter matrix. Represents the identity matrix; Obtained using system input and output data and use Update Eventually, the conditions will be met. Data written to In, or output those that meet the conditions As a control gain; in, Represents the parameter matrix The value of the (k+1)th iteration. Represents the parameter matrix The value of the kth iteration. Represents the parameter matrix The value of the kth iteration. Indicates the parameter value; Represents the parameter matrix The value of the kth iteration. Represents the parameter matrix The value of the (k+1)th iteration; First, determine Does it belong to a predefined bounded set? ; If it does not belong, then Reset to initial value If it belongs to the category, then determine... and Does the norm of the difference satisfy the requirements? , This represents a predefined normal parameter; if satisfied, output the result. As the control gain K; If the condition is not met, then the condition that is met will be... Data written to middle.

3. The smart grid intrusion-tolerant safety tracking and control method according to claim 2, characterized in that, In step 6, the event triggers the controller input of the elastic controller. Represented as: ； in, A flag indicating that an attack has occurred; ； ； This represents the k-th trigger time of distributed generator i. This represents the k'-th trigger time of distributed generator j. This is the moment of the last successful data transmission before a DoS attack occurs.

4. The smart grid intrusion-tolerant security tracking and control method according to claim 3, characterized in that, In step 6, the voltage tracking control task proceeds as follows: By comparing the voltage status information of its own generator in the event-triggered elastic controller with the voltage status information received from neighboring generators and the reference voltage information, the voltage status information of its own generator is continuously adjusted to reduce the difference between its own generator voltage and the reference voltage, thereby completing the voltage tracking control task.

5. A reinforcement learning-based smart grid intrusion tolerance and security tracking control system for implementing the reinforcement learning-based smart grid intrusion tolerance and security tracking control method as described in claim 1, characterized in that, The smart grid intrusion tolerance and security tracking control system based on reinforcement learning includes: The dynamic model building module is used to establish a dynamic model for generator secondary voltage tracking control in a microgrid system. The attack model construction module establishes a malicious network attack model for DoS attacks to describe the mechanism of DoS attacks and their impact on smart grid systems. Based on the mechanism of DoS attacks and their impact, it provides piecewise functions representing the occurrence and non-occurrence of attacks, as well as the assumptions about the duration and frequency of attacks that a denial-of-service attack must satisfy. The event trigger building module designs event triggers to send the voltage value of each distributed generator to its connected neighboring distributed generators and transmit the corresponding data to the fully distributed controller designed below. The control gain module calculates the control gain of the reinforcement learning-based controller by selecting a positive constant. The controller building module is designed with an attack compensation mechanism to resist the impact of network attacks on the system while conserving network communication resources. The voltage tracking control module, based on the control gain and the designed attack compensation mechanism, combined with the designed event trigger, obtains an event-triggered elastic controller to realize the voltage tracking control task.

6. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements the steps of the smart grid intrusion-tolerant security tracking control method based on reinforcement learning as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the smart grid intrusion-tolerant security tracking control method based on reinforcement learning as described in any one of claims 1 to 4.

Images