A smart grid false data injection attack defense method based on two-person zero-sum game

Abstract

The present application provides a kind of false data injection attack defense method of smart grid based on two-person zero-sum game, comprising: step one: build smart grid network structure model;Step two: build false data injection attack attack-defense model;Step three: build interactive scene and task space;Step four: based on deep deterministic policy gradient algorithm, realize optimal defense strategy solution.The present application models false data injection attack and defense behavior in smart grid based on two-person zero-sum game theory, sets reward function, and solves approximate Nash equilibrium point in game through deep deterministic policy gradient algorithm, to obtain the best defense strategy.The present application sets modeling as continuous process, compared with discrete process, can better describe the propagation of malicious software in the real world, and the deep deterministic policy gradient algorithm used can effectively solve the two-person zero-sum game problem with continuous action space.

Description

TECHNICAL FIELD

[0001] The present application relates to a smart grid false data injection attack defense method based on a two-person zero-sum game, belonging to the technical field of network space security. BACKGROUND

[0002] With the advancement of smart grid technology, the integration of its physical layer and information layer is increasingly deepening, making the boundary between the two no longer obvious. This technological progress indeed promotes significant efficiency improvement, however, it also leads to more severe challenges for grid security, including the increasing risk of various network attacks.

[0003] In the field of network security, false data injection attacks (FDIA) have attracted much attention due to their concealment and great destructive power, posing a serious threat to the security of smart grids and becoming one of the research hotspots. As a new type of network threat, FDIA manipulates data in smart grids, severely affecting the reliability and performance of the grid, and may lead to serious consequences and security risks. For example, FDIA can mislead the smart grid system by injecting false data, leading to power supply and demand imbalance and power supply interruption. In addition, FDIA may also cause errors in the control system of the smart grid, thereby affecting the safe and reliable supply of energy.

[0004] The essence of false data injection attack and defense can be seen as both sides making decisions that are most beneficial to themselves under limited resources, and the attack and defense strategies will also influence each other and dynamically change. In this process, the two sides are undoubtedly antagonistic, and the increase in one side's benefits means the decrease in the other side's benefits, which exactly meets the basic characteristics of a two-person zero-sum game. This game model assumes that there is an antagonistic relationship between the attacker and the defender, i.e., one side's gain is the other side's loss. By constructing this game model, the defense side can more effectively predict the attacker's strategy and design the optimal defense scheme to maximize its own interests. In addition, zero-sum game can also help identify and analyze potential attack targets, so that the defender can more accurately allocate resources, thereby improving the security and robustness of the entire smart grid system.

[0005] Malware propagation is a typical attack mode of false data injection attack, and many scholars have modeled and analyzed this process to study the physical information system security problems caused by false data injection attack in smart grid. Existing researches often model malware propagation as a discrete process, using methods based on discrete Markov process or probability theory to simulate propagation at discrete time points. However, considering the continuity of the real world, such modeling method may not truly reflect the continuous propagation process of malware in the real world. The Susceptible-Infectious-Susceptible (SIS) model is a mathematical model used to simulate the spread of infectious diseases in a population, and can also be used to simulate the propagation of malware in information systems. The differential equation of the continuous SIS model can be used to describe the transition of a target node between different states in smart grid, and the state of the node represents whether the node has been attacked by malware.

[0006] When solving the game problem, the continuous action space means that some conventional algorithms for solving the Nash equilibrium point cannot be used, because the infinite action space will cause dimension explosion, and the conventional method based on Q table algorithm cannot store the infinite continuous action space. The Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm for solving the problem of continuous action space. It combines deep learning and policy gradient method, and can effectively optimize the strategy in high-dimensional and continuous action space.

[0007] Based on the above technology, in order to solve the optimal defense strategy for false data injection attack in smart grid and reasonably schedule defense resources, the present application innovatively models malware propagation as a continuous process based on SIS model, and uses the deep deterministic policy gradient algorithm to solve the two-person zero-sum game problem with continuous action space, providing a new idea for defending against false data injection attack. SUMMARY

[0008] The technical problem solved by the present application is that the modeling of false data injection attack in smart grid is mostly discrete, which cannot truly reflect the reality. The present application models the false data injection attack and defense behavior in smart grid based on two-person zero-sum game theory, sets the reward function, and solves the approximate Nash equilibrium point in the game through the deep deterministic policy gradient algorithm, thereby obtaining the best defense strategy.

[0009] The technical solution adopted by the present application is: a smart grid false data injection attack defense method based on two-person zero-sum game, which comprises the following steps:

[0010] Step 1: Construct a smart grid network structure model

[0011] First, the network topology of the smart grid is modeled. Assuming the network has N bus nodes, its network topology is modeled as G =<V,E> Where V is the set of bus nodes (|V|=N), and E is the set of communication channels between them. An adjacency matrix can represent the connection relationships between nodes in a graph. Let M=(α ij ) N×N Let α represent the adjacency matrix of G, where α is the adjacency matrix of G such that if there is a communication channel between the i-th and j-th bus nodes. ij =1, otherwise α ij =0.

[0012] Step 2: Construct a fake data injection attack and defense model

[0013] S2.1 Malware Propagation Process Modeling

[0014] After establishing a network topology model of the smart grid, the propagation process of malicious data injection attacks is modeled based on this model. In any time slot, each bus node is in one of two states: "normal state" and "infected state." The former indicates that the bus node is not infected with malicious data injection attacks in that time slot, while the latter indicates that it is infected. A binary variable x is used... i,t Let x represent the state of the i-th bus node in the t-th time slot. i,t = 0 and 1 represent the normal and infected states of the bus, respectively. Based on this, vector X is used... t =(x i,t ,…,x N,t Let S represent the state of the bus node network in the t-th time slot. i,t and I i,t Let S represent the probabilities that the i-th bus node is in a normal state and an infected state in the t-th time slot, respectively. i,t and I i,t It is related (i.e., S) i,t +I i,t =1), therefore, the overall communication network state of the smart grid in the t-th time slot is represented as I. t =(I i,t ,…,I N,t ).

[0015] In information systems, malware can often be removed by deploying antivirus programs. Therefore, the state of a bus node is affected by both the propagation rate of fake data malware and the remediation rate of antivirus programs. A bus node i with a normal state might experience a β-level change in its state. jThe probability of infection of bus node i by false data malware from its adjacent infected bus node j. Similarly, an infected bus node i can be recovered to normal state by an antivirus program with a recovery rate of γ i .

[0016] The transition of bus nodes between these two states is represented by a differential system:

[0017]

[0018] where, I1is the initial state of the system, and T is the total number of time slots.

[0019] S2.2 Set the reward function of both attack and defense

[0020] In order to establish the game model, it is also necessary to set the reward function for both attack and defense. It is assumed that Q A and Q D represent the reward functions of the attacker and the defender, respectively. Since it is a zero-sum game for two people, Q A (β,γ)+Q D (β,γ) = 0 always holds. This means that during the game process, the expected benefit of the attacker is also the expected loss of the defender, which is composed of three parts: the effect of the attack expected to be destroyed Loss exp (β,γ), the cost of the attack Cost attack (β) of the attacker and the cost of the defense Cost defense (γ) of the defender. For the damage effect of the attack, it is defined as the amount of load shedding in the smart grid. When the false data malware successfully attacks a node, this node will return false measurement results to the control center, causing the control center to misjudge the whole system situation and re-analyze the power flow, resulting in the loss of a part of the load. The purpose of the attacker is to maximize the amount of load shedding. In addition, the attacker cost function and the defender cost function are additionally set, wherein the cost of the attacker is determined by the behavior β of the attacker, and similarly, the cost of the defender is determined by the behavior γ of the defender.

[0021] Therefore, the revenue function of the attacker is:

[0022] Q A (β,γ) = Loss exp (β,γ) - Cost attack (β) + Cost defense (γ)

[0023] Since it is a zero-sum game, the revenue function of the defender is:

[0024] Q D (β,γ) = -Loss exp (β,γ) + Costattack (β)-Cost defense (γ)

[0025] S2.3 Two-person zero-sum game modeling

[0026] A two-person zero-sum game is modeled. In the game, the objective of the attacker is to maximize its net benefit, while the objective of the defender is to minimize its total loss. The decision variables of the attacker and the defender are the malware propagation rate β and the repair rate γ at each bus node, respectively. In the real world, the values of these two rates depend on the attack resources and the defense resources, and how they allocate resources among the bus nodes.

[0027] Based on the proposed false data malware propagation and attack model, a two-person zero-sum game is modeled for the FDIA attacker and the defender in the smart grid, which is defined as a six-tuple as follows:

[0028] <N player ,X,A,D,Q A ,Q D >

[0029] where N player ={Att, Def} represents the FDIA attacker and defender agents participating in the game. In the real world, there can be more than one attacker and more than one defender, but to simplify unnecessary assumptions, all attackers and all defenders are considered as a whole in the present invention. Therefore, they are simplified as one attacker and one defender. X = (x1,..., x N ) represents the state of the bus node network in the smart grid, and the binary variable x i indicates the state of the i-th bus node, where x i = 0 and 1 represent the normal state and the infected state of the bus node, respectively. N is the total number of bus nodes in the smart grid. A and D represent the action sets of the attacker and the defender, respectively, and it is assumed that the attack action of the attacker is β = [β1,..., β N ], where β i represents the propagation rate of malware on the i-th bus node, and the defense action of the defender is γ = [γ1,..., γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i satisfy γ min ≤ γ i ≤ γ max , β min ≤ β i ≤ β max , i.e., the propagation rate and the repair rate should be between the upper bound and the lower bound. Q A and QD represent the reward functions of the attacker and defender defined in the last section, respectively.

[0030] Step three: build interactive scenarios and task spaces

[0031] S3.1 Define state space and action space

[0032] In order to build the interactive scenarios and task spaces of the agent interaction, it is necessary to define the state space and action space at the code level first. Based on the modeling of the two-person zero-sum game in the last step, the state space is defined as an N-dimensional binary variable X = (x1, …, xN), and the action space is defined as two continuous variables β = [β1, …, βN] and γ = [γ1, …, γN]. Since the expressions of the transmission rate and the repair rate are mathematical probabilities, the action space should satisfy 0 ≤ γi < 1, 0 ≤ βi < 1, i = 1, 2, …, N. In general, each agent of the attacker and the defender has its own independent action space, but shares a same state space, which also conforms to the attack and defense scenarios in the smart grid. N N N min i max min i max In general, each agent of the attacker and the defender has its own independent action space, but shares a same state space, which also conforms to the attack and defense scenarios in the smart grid.

[0033] The present application uses several common space types to declare and define the state space and task space. For an N-dimensional binary variable X = (x1, …, xN), a multi-dimensional binary variable is used to define it, and the value of each dimension can only be 0 or 1, indicating whether a certain bus node in the smart grid has been infected. For two continuous variables β = [β1, …, βN] and γ = [γ1, …, γN], first, a multi-dimensional continuous variable is used to define them respectively, which defines a continuous multi-dimensional space within a given range, and then a dictionary type variable is used to combine the two subspaces into a dictionary structure. The action space here is composed of two subspaces, one representing the “attacker” (Attacker) and the other representing the “defender” (Defender). N N N

[0034] S3.2 Implement the simulation power flow calculation function

[0035] ​​​​​​​​​​​The effect of attack and destruction Loss(β,γ) is an important part of the reward function of the attack and defense sides in the modeling stage, and the present application defines it as the cut load in the smart grid. Therefore, in order to quantitatively calculate the load loss caused before and after the attack, an interface Load(X) needs to be implemented when building the interactive scene, which is used to calculate the current load situation for a certain state X of the smart grid.

[0036] In this step, first, the power system model is created and the node parameters are initialized, including active load PD=(pd1,...,pd N ), reactive load QD=(qd1,...,qd N ), voltage amplitude VM=(vm1,...,vm N ), etc. Then, according to the current state X, the active load value pd i of the infected node i in the power system model is set to 0, which simulates the situation that the load of the node is attacked or fails. Finally, the optimal power flow calculation is performed in the DC power flow calculation mode, and the result load X is returned.

[0037] S3.3 Building an interactive scene

[0038] After the first two steps, the interactive scene needs to implement the state transition function

[0039] Trans(X,β,γ), which is used to output the new state X' and the reward functions Q A and Q D of the attack and defense sides given the current state X and the actions β and γ of the attack and defense sides.

[0040] The logic of the state transition part is based on the SIS model. For each node i, if its state variable x i =1, i.e., the node has been infected by malware, a random number rand i between 0 and 1 is randomly generated using a random function, and if rand i < γ i , the node is successfully repaired, and x i '=0 is obtained; if its state variable x i =0, i.e., the node has not been infected by malware, for each adjacent infected node j that satisfies α ij =1 and x j =1, a random number rand i,j between 0 and 1 is generated, and if rand i,j < β i , the node is successfully infected, and x i '=1 is obtained, otherwise x i '=0.

[0041] For the reward function, define:

[0042] Q A = Load(X') - Load(X) - Cost attack (β) + Cost defend (γ), where Cost attack (β) and

[0043] Cost defend (γ) are the attack and defense costs, which can be defined using an inverse proportional function and where k1 and k2 are the cost parameters for attack and defense. Due to the zero-sum nature of the game, there is also Q B = -Q A .

[0044] Based on the above definitions, the interactive environment is built as Env = {X, Trans(X, β, γ)}.

[0045] Step four: Implementing the optimal defense strategy based on the deep deterministic policy gradient algorithm

[0046] S4.1 Define the DDPG agent

[0047] Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows:

[0048] First, define the network structure and parameter update method of the DDPG agent. The Buffer class is responsible for storing the experience of the agent interacting with the environment, in the form of a tuple {X, X', β, Q)}. The Buffer class has a fixed capacity, and if the capacity is full when new experience is pushed to the Buffer, the oldest experience will be overwritten.

[0049] Actor is a feedforward neural network that receives the current state X and outputs an action β. The network contains three hidden layers with different numbers of neurons, and the final output action is scaled to [β min , β max ] or [γ min , γ max ].

[0050] Critic is also a feedforward neural network that receives the current state X and action β, and outputs the corresponding Q value, i.e., the value of action β in the current state X. Critic is similar to Actor, also with three hidden layers, but the input is the concatenation of state and action.

[0051] The DDPG agent includes a Buffer class, an Actor network and a Critic network, and their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process.

[0052] In addition, the DDPG agent provides a parameter update method Update() for the Actor and Critic networks, which is responsible for updating the Actor and Critic networks and performing soft updates on the target networks Actor_t and Critic_t.

[0053] S4.2 Initialize and train the attack and defense agents

[0054] Before starting the training, first define the environment Env and the Attacker and Defender agents, which represent the attack and defense sides, respectively. When using the DDPG algorithm to solve the two-person zero-sum game problem, the key is to ensure that both sides converge to the optimal solution. The DDPG algorithm itself is designed to solve the reinforcement learning problem in continuous action space, but when dealing with two-person zero-sum games, some specific adjustments and optimization strategies are needed in the training method. In order to make both sides take the better solution of their own, the present invention chooses the training strategy of alternating training of attack and defense sides.

[0055] Specifically, the steps for training the attacker Attacker are as follows:

[0056] First, randomly initialize the environment state. For each bus node in the smart grid, randomly decide whether it is in the infected state, where X random is a randomly generated N-dimensional binary variable representing the initial state of the environment.

[0057] Env.X = X random

[0058] Second, the attack and defense sides select their actions. In order to introduce randomness in the strategy exploration and avoid falling into local optimum, the action selected by the attack side is additionally added with noise ξ.

[0059] β = Attacker.Actor(X) + ξ

[0060] γ = Defender.Actor(X)

[0061] Third, obtain the environment feedback. The environment Env outputs the new state X' and the reward functions Q A and Q D according to the current state X and the actions β and γ of the attack and defense agents.

[0062] X', Q A , B = Env.Trans(X, β, γ)

[0063] Fourth, update the attacker. Store the interaction process {X, X', β, Q A} into the Buffer of Attacker as one of the samples for later training. Call the update function Update() of Attacker to iterate the parameters of its Actor and Critic networks.

[0064] Attacker.update()

[0065] Fifth, repeat steps one to four for 10 times to complete the training of this round of Attacker.

[0066] The steps of training Defender are similar, except that noise ξ is added to the action γ of Defender when selecting action, and the parameters of the Actor and Critic networks of Defender are iterated when updating parameters. The training of Attacker and Defender is alternated, and continues for multiple rounds until the loss functions of both sides converge.

[0067] The training process of each round of Attacker and Defender is shown in Figure 1 The upper half of the figure represents the two agents Attacker and Defender, including their respective network parameter updating methods Update(), experience pool Buffer, Actor network, Critic network, and corresponding target networks Actor_t and Critic_t. The lower half of the figure represents the interaction environment Env, the main part of which is the created power system model. In the figure, the IEEE14 bus node system is taken as an example, where each number represents a bus node, and five nodes 1, 2, 3, 6, and 8 have generators, and the arrows represent that the active power of the node is not zero.

[0068] S4.3 Solve the optimal defense strategy

[0069] After the training of Attacker and Defender is completed, both sides of the attack and defense agents can output the corresponding attack strategy β and defense strategy γ when given a power network state X, so that the reward function of itself is maximized. Therefore, only need to input the current state X into the Actor network of Defender, the approximate optimal defense strategy γ * can be obtained.

[0070] γ * = Defender.Actor(X)

[0071] The advantages of the present application compared with the prior art are:

[0072] (1) The present application models the malware propagation as a continuous process based on the SIS model, which can more accurately capture the state changes of malware propagation at any time. This means that a finer time granularity and higher accuracy can be obtained without being limited by fixed time steps. This modeling approach can more accurately describe and analyze the propagation dynamics of malware, especially in scenarios that require fine time granularity and complex dynamic behavior analysis, which has obvious advantages.

[0073] (2) The present application introduces the deep deterministic policy gradient algorithm, which combines deep learning and policy gradient method, can effectively optimize the policy in high-dimensional, continuous action space, and directly output the Q value of the action through the policy network without discretizing the continuous action space or complex action sampling, solving the dimension explosion problem caused by the infinite action space. At the same time, deterministic policy is used instead of random policy for updating, which can more efficiently optimize the policy in high-dimensional continuous action space. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a schematic diagram of a feature gray image.

[0075] The symbols in the figure are explained as follows:

[0076] X represents the current environment state;

[0077] β and γ represent the actions of Attacker and Defender respectively;

[0078] X' represents the new environment state after the actions of both sides;

[0079] Q A and Q B represent the reward functions of Attacker and Defender respectively. DETAILED DESCRIPTION

[0080] The intelligent power grid false data injection attack and defense method based on two-person zero-sum game proposed by the present application needs to solve the following three problems: (1) how to model the malware propagation as a continuous process; (2) how to set the reward functions of both sides; (3) how to apply the deep deterministic policy gradient algorithm to make it more suitable for the present application.

[0081] The main implementation idea of the application is: the application models false data injection attack and defense behavior in smart grid based on two-person zero-sum game theory, sets a reward function, and solves the approximate Nash equilibrium point in the game through deep deterministic policy gradient algorithm, and then obtains the best defense strategy. First, the propagation process of false data injection attack malware is modeled based on SIS propagation model, and the time granularity is further refined and the accuracy is improved. Then, based on the two-person zero-sum game theory, the reward function of the attack and defense sides is divided into three parts of attack effect, attack cost and defense cost, and the zero-sum nature in the game is fully utilized. Then, the Matpower simulation platform and OpenAI Gym are used to build the interaction scene and task space, which can perform real-time power flow calculation on any state of the power network to determine the attack effect. Finally, the DDPG algorithm is used to solve the dimension explosion problem caused by the infinite action space, and the strategy optimization is efficiently and stably performed to solve the best defense strategy.

[0082] The application proposes a false data injection attack and defense method for smart grid based on two-person zero-sum game, and the specific implementation steps of the application are described below:

[0083] Step one: build a smart grid network structure model

[0084] First, the network topology structure of the smart grid is modeled, and the network is assumed to have N bus nodes, then the network topology structure is modeled as G = <V, E>, where V is the set of bus nodes (|V| = N), and E is the set of communication channels between them. The adjacency matrix can represent the connection relationship of the nodes in the graph, and let M = (α ij ) N×N is the adjacency matrix of G, where if there is a communication channel between the i-th and j-th bus nodes, then α ij = 1, otherwise α ij = 0.

[0085] Take the 14-bus node system given by the Institute of Electrical and Electronics Engineers (IEEE) as an example, at this time the number of bus nodes N is 14, and the adjacency matrix M = (α ij ) 14×14 takes the following values:

[0086]

[0087] Step two: build an attack and defense model for false data injection attack

[0088] S2.1 Malware propagation process modeling

[0089] After establishing a network topology model of the smart grid, the propagation process of malicious data injection attacks is modeled based on this model. In any time slot, each bus node is in one of two states: "normal state" and "infected state." The former indicates that the bus node is not infected with malicious data injection attacks in that time slot, while the latter indicates that it is infected. A binary variable x is used... i,t Let x represent the state of the i-th bus node in the t-th time slot. i,t = 0 and 1 represent the normal and infected states of the bus, respectively. Based on this, vector X is used. t =(x i,t ,...,x N,t Let S represent the state of the bus node network in the t-th time period. i,t and I i,t Let S represent the probabilities that the i-th bus node is in a normal state and an infected state in the t-th time slot, respectively. i,t and I i,t It is related (i.e., S) i,t +I i,t =1), therefore, the overall communication network state of the smart grid in the t-th time slot is represented as I. t =(I i,t ,...,I N,t ).

[0090] In information systems, malware can often be removed by deploying antivirus programs. Therefore, the state of a bus node is affected by both the propagation rate of fake data malware and the remediation rate of antivirus programs. A bus node i with a normal state might experience a β-level change in its state. j The probability of infection originating from a neighboring, infected bus node j is γ, which is infected with fake data malware. Similarly, an infected bus node i can be restored to a normal state by an antivirus program, with a recovery rate of γ. i .

[0091] The transition of the bus node between these two states is represented by a differential system. Let ε represent a very small time interval, and the probability that any normal bus node is infected with fake data malware at time t+ε is estimated as follows:

[0092]

[0093] According to the theory of continuous-time Markov chain, this probability estimation will become accurate when ε→0. Therefore, this formula accurately evaluates the probability of a normal bus node being infected with false data malware. The estimation model first calculates the probability of a normal bus node being infected with false data malware from adjacent bus nodes that have already been infected. On this basis, by taking the sum of these probabilities, the exact probability of a normal bus node being infected with false data malware is calculated.

[0094] Similarly, the probability estimation of any infected bus node being removed of false data malware at the next time period t+ε is:

[0095] Pr{x i,t+ε =0|x i,t =1}=εγ j

[0096] Therefore, the probability estimation of any normal bus node remaining in its normal state and the probability of any infected bus node remaining in its infected state are:

[0097]

[0098] Pr{x i,t+ε =1|x i,t =1}=1-εγ j

[0099] By combining all the above formulas, the total probability formula can be obtained:

[0100]

[0101] This formula can also be written as:

[0102]

[0103] By making ε→0, the following differential system can be used to describe the state evolution of the bus node network in the smart grid:

[0104]

[0105] where I1 is the initial state of the system, and T is the total number of time slots.

[0106] S2.2 Set the reward functions of both attack and defense

[0107] In order to establish the game model, it is also necessary to set the reward functions for both attack and defense. It is assumed that Q A and Q D represent the reward functions of the attacker and the defender, respectively. Due to the zero-sum nature in two-person zero-sum game, it can be obtained that Q A (β,γ)+Q D(β, γ) = 0 is always true. This means that the expected benefit of the attacker, which is also the expected loss of the defender, is the effect of the attack Loss exp (β, γ), the cost of the attack Cost attack (β) and the cost of the defense Cost defense (γ) are three parts. For the effect of the attack, it is defined as the amount of load shedding in the smart grid. When the false data malware successfully attacks a node, the node will return false measurement results to the control center, causing the control center to misjudge the situation of the entire system and re-analyze the power flow, resulting in the loss of a part of the load. The purpose of the attacker is to maximize the amount of load shedding. In addition, an attacker cost function and a defender cost function are additionally set, wherein the cost of the attacker is determined by the behavior of the attacker β, and similarly, the cost of the defender is determined by the behavior of the defender γ.

[0108] Therefore, the revenue function of the attacker is:

[0109] Q A (β, γ) = Loss exp (β, γ) - Cost attack (β) + Cost defense (γ)

[0110] Since it is a zero-sum game, the revenue function of the defender is:

[0111] Q D (β, γ) = -Loss exp (β, γ) + Cost attack (β) - Cost defense (γ)

[0112] S2.3 Two-person zero-sum game modeling

[0113] Two-person zero-sum game modeling is performed. In the game, the goal of the attacker is to maximize its net benefit, while the goal of the defender is to minimize its total loss. The decision variables of the attacker and the defender are the malware propagation rate β and the repair rate γ on each bus node, respectively. In the real world, the values of these two rates depend on the attack resources and defense resources, and how they allocate resources among bus nodes.

[0114] Based on the proposed false data malware propagation and attack model, two-person zero-sum game modeling is performed for the FDIA attacker and the defender in the smart grid, which is defined as the following six-tuple:

[0115] <N player ,X,A,D,Q A ,Q D >

[0116] where N player ={Att,Def} represents the FDIA attacker and defender agents participating in the game. In the real world, there can be more than one attacker and more than one defender, but in order to simplify unnecessary assumptions, all attackers and all defenders are considered as a whole in this invention. Therefore, they are simplified as an attacker and a defender. X=(x1,...,x N ) represents the state of the bus node network in the smart grid, and the binary variable x i represents the state of the i-th bus node, where x i =0 and 1 represent the normal state and the infected state of the bus node, respectively. N is the total number of bus nodes in the smart grid. A and D represent the action sets of the attacker and the defender, respectively, assuming that the attack operation of the attacker is β=[β1,...,β N ], where β i represents the propagation rate of malware on the i-th bus node, and the defense operation of the defender is γ=[γ1,...,γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i satisfy γ min ≤γ i ≤γ max ,β min ≤β i ≤β max , i.e. the propagation rate and the repair rate should be between the upper and lower bounds. Q A and Q D represent the reward functions of the attacker and the defender defined in the previous section, respectively.

[0117] Step three: build interactive scenarios and task space

[0118] S3.1 Define state space and action space

[0119] In order to build the interactive scenario and task space of the agent interaction, first of all, the state space and action space need to be defined at the code level. Based on the modeling of the two-person zero-sum game in the previous step, the state space is defined as the N-dimensional binary variable X=(x1,...,x N ), and the action space is defined as two continuous variables β=[β1,...,β N ] and γ=[γ1,...,γ N ], since the propagation rate and the repair rate expressed are mathematical probabilities, the action space should satisfy 0≤γ min ≤γ i ≤γ max <1,0≤β min ≤βi ≤β max <1. In general, each agent on both the attacking and defending sides has its own independent action space, but shares the same state space, which is consistent with the attack and defense scenarios in smart grids.

[0120] OpenAI Gym is a toolkit for developing and comparing reinforcement learning algorithms. It provides a standard set of environment interfaces, allowing researchers and developers to easily build, train, and evaluate reinforcement learning models. Gym is widely used in academic research, industrial applications, and the learning process for developing RL algorithms. With flexible extensibility, Gym allows users to customize the environment to their needs, and even create their own problems by modifying or extending existing environments. It is compatible with various reinforcement learning libraries and deep learning frameworks, such as PyTorch and TensorFlow.

[0121] This paper uses several common space types provided by OpenAI Gym to declare and define the state space and task space. For a 14-dimensional binary variable X = (x1,...,x...) 14 The variable β is defined using the type spaces.MultiBinary, where each dimension can only have a value of 0 or 1, indicating whether a bus node in the IEEE 14 bus node system has been infected. For two consecutive variables β = [β1,...,β...],... 14 ] and γ = [γ1,...,γ 14 First, the `spaces.Box` type variable is used to define each subspace separately, which defines a continuous multidimensional space within a given range. Then, the `spaces.Dict` type variable is used to combine the two subspaces into a dictionary structure. Here, the action space consists of two subspaces, one representing the "Attacker" and the other representing the "Defender".

[0122] S3.2 implements the function of power flow simulation calculation.

[0123] When setting the reward functions for both the attacker and defender during the modeling phase, the loss effect of attack destruction is considered. exp (β,γ) is an important component, which is defined in this invention as the load shedding amount in the smart grid. Therefore, in order to quantitatively calculate the load loss before and after the attack, an interface Load(X) needs to be implemented when building the interactive scenario. Its function is to calculate the current load situation for a certain state X of the smart grid.

[0124] In this step, first, the loadcase() function provided by the simulation tool Matpower is passed in the parameter 'case14', creating an IEEE14 bus node power system model and initializing the node parameters, including active load PD=(pd1,..., pd 14 ), reactive load QD=(qd1,..., qd 14 ), voltage amplitude VM=(vm1,..., vm 14 ), etc. Then the active load value pd i of the infected node i in the power system model is set to 0 according to the current state X, which simulates the situation that the load of the node is attacked or fails. Finally, it is configured as a direct current flow calculation mode by mpoption(mpoption,'PF_DC',1), and the runopf() function is called to run the optimal power flow calculation, returning the result load X = Load(X) = Loss exp (β,γ).

[0125] S3.3 Build an interactive scene

[0126] After the first two steps, the interactive scene needs to implement the state transition function

[0127] Trans(X,β,γ), which outputs the new state X' and the reward functions Q A and Q D of the attacking and defending agents given the current state X and the actions β and γ of the attacking and defending agents.

[0128] The logic of the state transition part is based on the SIS model. For each node i, if its state variable x i = 1, i.e. the node has been infected by malware, then use random.random() to randomly generate a 0-1 random number rand i , if rand i < γ i , then the node is successfully repaired, x i '=0; if its state variable x i =0, i.e. the node has not been infected by malware, then for each adjacent infected node j that satisfies α ij =1 and x j =1, generate a 0-1 random number rand i,j , if rand i,j < β i , then the node is successfully infected, x i '=1, otherwise x i '=0.

[0129] For the reward function, define Q A = Load(X') - Load(X) - Cost attack (β) + Cost defend (γ), where Cost attack (β) and Cost defend (γ) are the attack and defense costs of both sides, which can be defined as and where k1 and k2 are the cost parameters of attack and defense. Since the game is zero-sum, there is also Q B = -Q A .

[0130] Based on the above definitions, the interactive environment is built as Env = {X, Trans(X, β, γ)}.

[0131] Step four: implement the optimal defense strategy solution based on the deep deterministic policy gradient algorithm

[0132] S4.1 Define DDPG agent

[0133] Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows:

[0134] First, define the network structure and parameter update method of the DDPG agent. The Buffer class is responsible for storing the experience of the agent interacting with the environment, in the form of a tuple {X, X', β, Q} respectively storing the old state, new state, action, and reward function. The capacity of the Buffer class is fixed, and if the new experience is pushed to the Buffer when the capacity is already full, the oldest experience will be overwritten.

[0135] Actor is a feedforward neural network that receives the current state X and outputs an action β. The network contains three hidden layers with different numbers of neurons, and the final output action is scaled to [β min , β max ] or [γ min , γ max ].

[0136] Critic is also a feedforward neural network that receives the current state X and action β, and outputs the corresponding Q value, i.e. the value of action β in the current state X. Critic is similar to Actor, also with three hidden layers, but the input is the concatenation of state and action.

[0137] The DDPG agent contains a Buffer class, an Actor network and a Critic network, as well as their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process.

[0138] In addition, the DDPG agent provides a parameter update method Update() for both the Actor and Critic networks, which is responsible for training the Actor and Critic networks and mainly includes the following steps:

[0139] First, a data sample {X, X', β, Q} is randomly sampled from the Buffer class, which is used to calculate the loss function of the network and perform gradient descent.

[0140] Second, the target Q value Target_Q is calculated. Target_Q is calculated by the target networks Actor_t and Critic_t, and is used to guide the learning of the Critic network. Specifically, first calculate the action β or γ corresponding to the next state through the Actor target network Actor_t, and then calculate the Q value Q of the action through the Critic target network Critic_t. future As shown below.

[0141] Q future = Critic_t(X', Actor_t(X'))

[0142] The target Q value Target_Q represents the discounted sum of the current reward and future rewards, as shown below, where α is the discount factor used to balance the importance of the current and future rewards.

[0143] Target_Q = Q + αQ future

[0144] Third, the current Q value Current_Q is calculated. Current_Q is calculated by the current Critic network, and represents the value of the agent performing the current action β in the current state X.

[0145] Current_Q = Critic(X, β)

[0146] Fourth, the Critic network is optimized. The loss function of the Critic network is the mean square error between the current Q value Current_Q and the target Q value Target_Q. The smaller the loss function, the more accurate the Critic network's estimate of the Q value.

[0147] After calculating the loss, the parameters of the Critic network are updated using the backpropagation algorithm.

[0148] Step 5: Optimize the Actor network. The loss function of the Actor network is defined as the negative value of the Q value calculated by the Critic network. This means that the goal of the Actor network is to find an action that maximizes the Q value output by the Critic network, thereby selecting the optimal policy.

[0149] After calculating the loss, the parameters of the Actor network are updated using the backpropagation algorithm.

[0150] Step 6: Update the target networks Actor_t and Critic_t. The parameters of Actor_t and Critic_t are updated using a soft update method. Specifically, the parameters of Actor_t and Critic_t are the weighted sum of the current Actor and Critic network parameters and their own parameters. The purpose of soft update is to make the parameter changes of the target network more smooth and stable, thereby improving the stability of the training process.

[0151] S4.2 Initialize and train the attacker and defender intelligent agents

[0152] Before starting training, first define the environment Env and the Attacker and Defender intelligent agents, representing the attacking and defending sides. When using the DDPG algorithm to solve the two-person zero-sum game problem, the key is to ensure that both sides converge to the optimal solution. The DDPG algorithm itself is designed to solve the reinforcement learning problem in continuous action space, but when dealing with two-person zero-sum games, some specific adjustments and optimization strategies are needed in the training method. In order to make both sides take the better solution for their own side, the invention chooses the training strategy of alternating training of the attacker and defender.

[0153] Specifically, the steps to train the attacker Attacker are as follows:

[0154] Step 1: Randomly initialize the environment state. For each bus node in the smart grid, randomly decide whether it is in an infected state.

[0155] Env.X = X random

[0156] Step 2: The attacker and defender select their respective actions. In order to introduce randomness in the strategy exploration and avoid falling into local optimization, additional noise ξ is added to the action selected by the attacker.

[0157] β = Attacker.Actor(X) + ξ

[0158] γ = Defender.Actor(X)

[0159] Third, get the environment feedback. The environment Env outputs a new state X' and the reward function Q of both sides according to the current state X and the actions β and γ of the attacking and defending sides A and Q D .

[0160] X', Q A , Q B = Env.Trans(X, β, γ)

[0161] Fourth, update the attacker. Store the interaction process {X, X', β, Q A} in the Buffer of the Attacker as one of the samples for later training. Call the update function Update() of the Attacker to iterate the network parameters of the Actor and Critic of the Attacker.

[0162] Attacker.update()

[0163] Fifth, repeat steps one to four for 10 times to complete the training of the attacker Attacker in this round.

[0164] The steps of training the defender Defender are similar, except that noise ξ is added to the defender action γ when selecting the action, and the network parameters of the Actor and Critic of the Defender are iterated when updating the parameters. The training of the Attacker and the Defender is alternated, and continues for multiple rounds until the loss functions of both sides converge.

[0165] The training process of the Attacker and the Defender in each round is shown in Figure 1 The upper half of the figure represents the two agents Attacker and Defender, including their respective network parameter update methods Update(), experience pool Buffer, Actor network, Critic network, and corresponding target networks Actor_t and Critic_t. The lower half of the figure represents the interaction environment Env, the main part of which is the created power system model. In the figure, the IEEE14 bus node system is taken as an example, and each number represents a bus node. The arrows represent that the node has active power output, and the letters G represent that there are generators in five nodes 1, 2, 3, 6, and 8.

[0166] In this step, the alternating training strategy is adopted because in a two-person zero-sum game, the strategies of both sides are changing. Because the goal of each side is to maximize its own reward and minimize the reward of the opponent, when one side updates the strategy, the other side will be directly affected. Alternating training ensures that both sides can adapt to the changes of the opponent in the process of strategy training.

[0167] If only one side is trained while the other is ignored, the trained strategy can only be applicable to the current opponent strategy. Once the opponent's strategy changes, the trained strategy can be invalid. Therefore, alternating training can help both sides to constantly adapt to the changes in the opponent's strategy, so as to find a more robust solution.

[0168] One of the core features of two-player zero-sum games is antagonism. When one side takes a certain strategy, the goal of the other side is to find the best counter-strategy to minimize the opponent's gain. In this antagonistic environment, alternating training can be seen as part of adversarial learning.

[0169] During the process of alternating training, the strategies of both sides are constantly improving, and through constantly responding to the optimal strategy of the opponent, the performance of both sides will gradually tend to be optimal. The mechanism of adversarial learning enables the agent to perform well even in the worst case, which is difficult to achieve in simple fixed training.

[0170] S4.3 Solving the optimal defense strategy

[0171] After the training of Attacker and Defender is completed, both agents can output the corresponding attack strategy β and defense strategy γ given a power network state X, so that the reward function of themselves is maximized. Therefore, only need to input the current state X into the Actor network of Defender, the approximate optimal defense strategy γ can be obtained * .

[0172] γ * = Defender.Actor(X)

[0173] In order to verify the effectiveness of the trained Defender agent in defending against false data injection attacks in the IEEE14 bus node system and the approximate optimality of the defense strategy γ * , the following experiments can be done:

[0174] Assume that the state of the IEEE14 bus node system is fixed X = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), and the attack strategy of the attacker Attacker is β = β * . Randomly select the defense strategy of the defender Defender 999 times γ1, γ2,..., γ 999 , respectively. Calculate the reward functions Q1, Q2,..., Q 999 , Q * of these random defense strategies and the approximate optimal defense strategy when the system state and the attack strategy are unchanged, and sort them in descending order according to the size of the reward function. The results are as follows.

[0175] Serial number Defence strategy Q value 1 (0.00,0.38,0.00,0.00,0.00,0.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00) -85.31 2 (0.24,0.41,0.01,0.62,0.33,0.05,0.15,0.12,0.48,0.56,0.30,0.26,0.18,0.04) -115.26 3 (0.44,0.07,0.40,0.27,0.30,0.06,0.18,0.30,0.58,0.20,0.19,0.15,0.04,0.20) -115.54 4 (0.24,0.13,0.09,0.47,0.09,0.11,0.13,0.18,0.63,0.33,0.14,0.49,0.17,0.48) -120.12 … … … 999 (0.89,0.80,0.86,0.88,0.83,0.88,0.32,0.82,0.40,0.42,0.03,0.69,0.79,0.02) -326.73 1000 (0.12,0.90,0.85,0.61,0.28,0.24,0.79,0.84,0.86,0.86,0.89,0.67,0.87,0.68) -359.74

[0176] In the above table, the defense strategy given in the first row is the approximate optimal defense strategy γ calculated by the Defender agent * It can be seen that among the 1000 random defense strategies, the reward function value corresponding to this strategy is the largest, indicating that this strategy is optimal relative to other defense strategies, which also proves the effectiveness of the Defender agent trained this time in defending against false data injection attacks in the IEEE14 bus node system.

[0177] The contents not described in detail in the specification of the present application belong to the prior art known to those skilled in the art.

[0178] The above only describes one preferred embodiment of the false data injection attack defense method for smart grid based on two-person zero-sum game of the present application. It should be noted that for those skilled in the art, without departing from the principles of the false data injection attack defense method for smart grid based on two-person zero-sum game of the present application, some improvements and refinements can also be made, which should also be considered as the protection scope of the false data injection attack defense method for smart grid based on two-person zero-sum game of the present application.

Claims

1. A method for defending false data injection attack in smart grid based on two-person zero-sum game, characterized in that, Comprising the following steps: Step one: Constructing the network topology structure model of smart grid: Set the network has N bus nodes, then model the network topology structure as G=<V, E>, where V is the set of bus nodes (|V|=N), E is the set of communication channels between them; the adjacent matrix represents the connection relationship of the nodes in the graph, set M=(α ij ) N×N The adjacent matrix of G, if there is a communication channel between the ith and jth bus nodes, then α ij =1, otherwise α ij =0; Step two: build a false data injection attack defense model, including: malware propagation process modeling, setting up attack and defense reward function and two-person zero-sum game modeling; the malware propagation process modeling includes: at any time slot, each bus node is in one of two states: "normal state" and "infected state"; use binary variable x i,t to represent the state of the i-th bus node at the t-th time slot; x i,t = 0 and 1 represent the normal state and infected state of the bus respectively; use vector X t = (x i,t ,...,x N,t ) to represent the state of the bus node network at the t-th time slot; A bus node i in normal state can be infected with false data malware from an adjacent, infected bus node j with probability β j ; likewise, a bus node i in infected state is recovered to normal state by an antivirus program with repair rate γ i ; Step three: build the interactive scene and task space, including: defining state space and action space, implementing simulation power flow calculation function and building interactive scene; the building interactive scene needs to realize state transition function Trans(X, β, γ), the function is to give the current state X and the action β and γ of the attacking and defending agents, output the new state X' and the reward function Q of the attacking and defending agents A and Q D ; The logic of the state transition part is based on the SIS model. For each node i, if its state variable x i =1, meaning the node has been infected with malware, then a random number (rand) between 0 and 1 is generated using a random function. i If rand i <γ i If the node is successfully repaired, then x is obtained. i If its state variable x = 0; i =0, meaning the node has not yet been infected by malware. Therefore, for every node satisfying α... ij =1 and x j For each infected node j adjacent to node j with a value of 1, generate a random number rand between 0 and 1. i,j If rand i,j <β i If the node is successfully infected, then x is obtained. i ′=1, otherwise x i ′=0; An interactive environment is built as Env={X, Trans(X, β, γ)}; Step four: solving the optimal defense strategy based on the deep deterministic policy gradient algorithm, including: defining the DDPG agent, initializing and training the attack and defense agents, and solving the optimal defense strategy; before starting the training, first define the environment Env, representing the two agents Attacker and Defender of the attack and defense parties; use the DDPG algorithm to solve the two-person zero-sum game problem; after training the Attacker and Defender agents, the attack and defense agents output the corresponding attack strategy β and defense strategy γ when given a power network state X, so that the reward function of each agent is maximized, that is, the approximate optimal defense strategy is obtained. 2.The method of claim 1, wherein the method is characterized in that: In step two, the malware propagation process modeling includes: setting S i,t and I i,t represent the probability of the i-th bus node being in normal state and infected state at the t-th time slot, respectively; since S i,t and I i,t are related, i.e., S i,t + I i,t = 1, the overall communication network state of the smart grid at the t-th time slot is represented as I t = (I i,t ,...,I N,t ); The transformation of bus nodes between the two states is represented by a differential system: ； Where I1 is the initial state of the system, and T is the total number of time slots. 3.The method of claim 2, wherein the method further comprises: In step two, the setting of the attack and defense reward functions includes: Due to the zero-sum nature of the two-person zero-sum game, we have Q A (β,γ)+Q D (β,γ) = 0 holds constantly; in the process of game, the expected benefit of the attacker, also the expected loss of the defender, is the effect of the attack expected to destroy Loss exp (β,γ), the cost of the attacker's attack Cost attack (β) and the cost of defender's defense Cost defense (γ) three parts; for the damage effect of attack Loss exp (β, γ), defined as the amount of load shedding in smart grid; when the false data malware successfully attacks a node, this node will return false measurement results to the control center, leading to the control center misjudging the whole system situation and reanalyzing the power flow, resulting in losing part of the load; the attacker's goal is to maximize the load shedding amount, in addition, the attacker cost function and the defender cost function are additionally set, wherein the cost of the attacker is determined by the behavior β of the attacker, and similarly, the cost of the defender is determined by the behavior γ of the defender; Therefore, the attacker's revenue function is: Q A (β,γ) = Loss exp (β,γ) = Cost attack (β) = Cost defense (γ) Since it is a zero-sum game, the defender's revenue function is: Q D (β,γ) = -Loss exp (β,γ) + Cost attack (β) - Cost defense (γ).

4. The method of claim 1 or 3, wherein the method is based on a two-person zero-sum game. In step two, the two-person zero-sum game modeling includes: The two-person zero-sum game modeling of the FDIA attacker and defender in the smart grid is defined as the following six-tuple: < N player X, A, D, Q A Q D > where N player ={Att,Def} represents the FDIA attacker and defender agents participating in the game; there can be more than one attacker and more than one defender, and all attackers and all defenders are defined as a whole; therefore, it is simplified to one attacker and one defender; X=(x1,...,x N ) represents the state of the bus node network in the smart grid, and the binary variable x i represents the state of the i-th bus node; N is the total number of bus nodes in the smart grid; A and D represent the operation sets of the attacker and the defender, respectively, and the attack operation of the attacker is β=[β1,...,β N ], where β i represents the propagation rate of malware on the i-th bus node, and the defense operation of the defender is γ=[γ1,...,γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i both satisfy: γ min ≤γ i ≤γ max ,β min ≤β i ≤β max , i.e., both the propagation rate and the recovery rate should be between the upper and lower bounds; Q A and Q D represent the reward functions of the attacker and the defender, respectively. 5.The smart grid false data injection attack defense method based on two-person zero-sum game according to claim 1, characterized in that: In step three, the definition of the state space and action space includes: The state space is defined as N-dimensional binary variables X = (x1,..., x N ), with each dimension of X only able to take on the value of 0 or 1, representing whether a certain bus node in the smart grid has been infected or not; the action space is defined as two continuous variables β = [β1,..., β N ] and γ = [γ1,..., γ N ], with 0 ≤ γ min ≤ γ i ≤ γ max < 1, 0 ≤ β min ≤ β i ≤ β max < 1; where variable β represents the action subspace of the attacker, and variable γ represents the action subspace of the defender. 6.The method of claim 1 or 5, wherein: In step three, the implementation of the simulation power flow calculation function includes: First, a power system model is created and each node parameter is initialized, including active power load PD = (pd1,..., pdn), reactive power load QD = (qd1,..., qdn), voltage amplitude VM = (vm1,..., vmn). N N N );​​ Then the active power load value pd of the infected node i in the power system model is set to 0 according to the current state X, and finally the optimal power flow calculation is performed in the direct current flow calculation mode, and the result load is returned. i Set to 0, finally in the direct current flow calculation mode Optimal power flow calculation, return result load X = Load (X) = Loss exp (β, γ). 7.The method of claim 6, wherein the method further comprises: In step three, in the building of the interactive scene, for the reward function Q A , the definition is: Q A = Load(X') - Load(X) - Cost attack (β) + Cost defend (γ), where Cost attack (β) and Cost defend (γ) are the cost of attack and defense for both sides, which are defined as and where k1and k2are the cost parameters of attack and defense; due to the zero-sum nature of the game, there are also Q D = -Q A . 8.The smart grid false data injection attack defense method based on two-person zero-sum game according to claim 1, characterized in that: In step four, the definition of the DDPG agent includes: Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows: First, define the network structure and parameter update method of the DDPG agent; the Buffer class is responsible for storing the experience of the agent interacting with the environment, in the form of a tuple {X, X', β, Q} to store the old state, new state, action, and reward function; The actor network contains three hidden layers with different numbers of neurons, and the final output action is scaled to [β min ,β max ] or [γ min ,γ max ]; Critic is also a feedforward neural network that receives the current state X and action β and outputs the corresponding Q value, i.e., the value of action β in the current state X; Critic also has three hidden layers, and the input is the concatenation of the state and action; The DDPG agent contains a Buffer class, an Actor network, and a Critic network, as well as their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process; The DDPG agent provides a parameter update method Update() for the Actor and Critic networks, which is responsible for updating the Actor and Critic networks and performing soft updates on the target networks Actor_t and Critic_t. 9.The smart grid false data injection attack defense method based on two-person zero-sum game according to claim 8, characterized in that: In step four, the initialization and training of the attack and defense agents includes: The steps to train the attacker Attacker are as follows: First, the initial state of the environment is randomly initialized. For each bus node in the smart grid, a random decision is made as to whether it is in an infected state or not, where X random is a randomly generated N-dimensional binary variable representing the initial state of the environment. Env. X = X random Second, both sides choose their actions; in order to introduce randomness in the strategy exploration, avoid falling into local optimum, the attacker selected action will be added additional noise ξ; β = Attacker.Actor(X) + ξ γ = Defender.Actor(X) Third step, get the environmental feedback; the environment Env outputs the new state X' and the reward function Q of the offense and defense agents according to the current state X and the actions β and γ of the offense and defense agents A and Q D ; X', Q A Q D = Env.Trans(X, β, γ) Fourthly, update the attacker; store the interaction process {X, X', β, Q A} into the Buffer of the Attacker as one of the samples for later training; call the update function Update() of the Attacker to iterate the network parameters of the Actor and the Critic. Fifth, repeat the first to the fifth step 10 times, complete this round of training of the attacker Attacker; The steps of training the defender Defender are different from those of training the attacker Attacker. When selecting actions, noise ξ is added to the defender's action γ. When updating parameters, the parameters of the Actor and Critic networks of Defender are iterated. The training of Attacker and Defender is alternated, and continues for multiple rounds until the loss function of both sides converges.

Images