Intelligent power grid false data injection attack defense method based on two-person zero-sum game

Abstract

The invention provides a smart power grid false data injection attack defense method based on a two-person zero sum game. The method comprises the following steps: step 1, constructing a smart power grid network structure model; 2, constructing a false data injection attack attack and defense model; 3, establishing an interaction scene and a task space; and step 4, realizing optimal defense strategy solution based on a depth deterministic strategy gradient algorithm. According to the method, the false data injection attack and defense behavior in the smart power grid are modeled based on the two-person zero-sum game theory, a reward function is set, an approximate Nash equilibrium point in the game is solved through a depth deterministic strategy gradient algorithm, and then the optimal defense strategy is obtained. According to the method, modeling is set to be a continuous process, compared with a discrete process, propagation of malicious software in the real world can be better described, and meanwhile the two-person zero-sum game problem with the continuous action space can be effectively solved through the used depth deterministic strategy gradient algorithm.

Description

Technical Field

[0001] The invention relates to a method for defending against false data injection attacks of a smart grid based on a two-person zero-sum game, and belongs to the technical field of cyberspace security. Background Art

[0002] With the advancement of smart grid technology, the integration of its physical layer and information layer has become increasingly profound, making the boundary between the two less obvious. This technological progress has indeed promoted a significant improvement in efficiency, however, it has also led to more severe challenges to grid security, including the increasing risk of various types of cyber attacks.

[0003] In the field of network security, false data injection attacks (FDIA) have attracted much attention due to their concealment and great destructiveness. They pose a serious threat to the security of smart grids and have become one of the research hotspots. As a new type of network threat, FDIA seriously affects the reliability and performance of the smart grid by manipulating data in the smart grid, which may lead to serious consequences and security risks. For example, FDIA can mislead the smart grid system by injecting false data, resulting in an imbalance in power supply and demand and power outages. In addition, FDIA may also cause errors in the control system of the smart grid, thereby affecting the safe and reliable supply of energy.

[0004] The essence of false data injection attack and defense can be seen as the attacker and defender making the most favorable decision for themselves under limited resources. At the same time, the offensive and defensive strategies will also influence each other and change dynamically. In such a process, the attacker and defender are undoubtedly adversarial. The increase in the benefits of one party must mean the decrease in the benefits of the other party, which just meets the basic characteristics of a two-person zero-sum game. This game model assumes that there is an adversarial relationship between the attacker and the defender, that is, the gain of one party is the loss of the other party. By constructing this game model, the defender can more effectively predict the attacker's strategy and design the optimal defense plan to maximize its own interests. In addition, zero-sum games can also help identify and analyze potential attack targets, allowing defenders to allocate resources more accurately, thereby improving the security and robustness of the entire smart grid system.

[0005] Spreading through malware is a typical attack method of false data injection attack. Many scholars have modeled and analyzed this process and studied the physical information system security issues brought about by false data injection attacks in smart grids. Existing studies often model malware propagation as a discrete process, using discrete Markov process or probability theory-based methods to simulate the propagation at discrete time points. However, considering the continuity of the real world, such a modeling method may not truly reflect the continuous propagation process of malware in the real world. The Susceptible-Infectious-Susceptible (SIS) model is a mathematical model used to simulate the spread of infectious diseases in a group, and can also be used to simulate the spread of malware in information systems. The continuous SIS model can be used to establish differential equations to describe the transition between different states of a target node in the smart grid. The state of the node represents whether the node has been attacked by malware.

[0006] When solving game problems, continuous action space means that some conventional algorithms for solving Nash equilibrium points can no longer be used. This is because infinite action space will lead to dimensionality explosion, and the Q-table-based algorithm in conventional methods cannot store infinite continuous action space. The Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm for solving continuous action space problems. It combines deep learning and policy gradient methods to perform effective policy optimization in high-dimensional, continuous action space.

[0007] Based on the above technology, in order to solve the optimal defense strategy against false data injection attacks in smart grids and reasonably schedule defense resources, the present invention innovatively models malware propagation as a continuous process based on the SIS model, and uses a deep deterministic policy gradient algorithm to solve the two-person zero-sum game problem with continuous action space, providing a new idea for defending against false data injection attacks. Summary of the invention

[0008] The technology of the present invention solves the problem: the modeling of false data injection attacks in smart grids is mostly discrete and cannot truly reflect the shortcomings of the real world. The present invention models false data injection attacks and defense behaviors in smart grids based on two-person zero-sum game theory, sets a reward function, and solves the approximate Nash equilibrium point in the game through a deep deterministic policy gradient algorithm, thereby obtaining the optimal defense strategy.

[0009] The technical solution adopted by the present invention is: a method for defending against false data injection attacks of smart grids based on two-person zero-sum game, which comprises the following steps:

[0010] Step 1: Construct a smart grid network structure model

[0011] First, the network topology of the smart grid is modeled. Assuming that the network has N bus nodes, its network topology is modeled as G =<V,E> , where V is the set of bus nodes (|V|=N), and E is the set of communication channels between them. The adjacency matrix can represent the connection relationship of nodes in the graph. Let M=(α ij ) N×N represents the adjacency matrix of G, where if there is a communication channel between the i-th and j-th bus nodes, then α ij =1, otherwise α ij =0.

[0012] Step 2: Build a false data injection attack and defense model

[0013] S2.1 Modeling of malware propagation process

[0014] After establishing the network topology model of the smart grid, the propagation process of the false data injection attack malware is modeled on its basis. In any time slot, each bus node is in one of two states: "normal state" and "infected state". The former indicates that the bus node is not infected with false data malware in this time slot, and the latter indicates that it is infected with false data malware. Using the binary variable x i,t To represent the state of the i-th bus node in the t-th time slot. i,t = 0 and 1 represent the normal state and infection state of the bus respectively. On this basis, use the vector X t =(x i,t ,…,x N,t ) is used to represent the state of the bus node network in the tth time slot. Let S i,t and I i,t They represent the probability that the i-th busbar node is in a normal state and an infected state in the t-th time slot. i,t and I i,t is related (i.e. S i,t +I i,t =1), so the overall communication network state of the smart grid in the tth time slot is expressed as I t =(I i,t ,…,I N,t ).

[0015] In information systems, malware can usually be removed by deploying antivirus programs. Therefore, the state of the bus node is affected by both the propagation rate of fake data malware and the repair rate of antivirus programs. A bus node i with a normal state may be jThe probability of being infected by fake data malware from the adjacent, infected bus node j is γ. Similarly, the infected bus node i can be restored to normal by the antivirus program with a repair rate of γ i .

[0016] The transition of the busbar node between these two states is represented by a differential system:

[0017]

[0018] Among them, I 1 is the initial state of the system, and T is the total number of time slots.

[0019] S2.2 Setting the reward function for both attackers and defenders

[0020] In order to establish a game model, it is also necessary to set a reward function for both the attacker and the defender. Assume that Q A and Q D Represent the reward functions of the attacker and the defender respectively. Due to the zero-sum property in the two-player zero-sum game, we can get Q A (β,γ)+Q D (β,γ)=0 is always true. This means that in the game process, the expected benefit of the attacker is also the expected loss of the defender, which is the expected damage effect of the attack. exp (β,γ), the cost of the attacker's attack attack (β) and the defender's defense cost Cost defense The destructive effect of the attack is defined as the amount of load shedding in the smart grid. When the false data malware successfully attacks a node, the node will return an erroneous measurement result to the control center, causing the control center to misjudge the situation of the entire system and re-analyze the power flow, resulting in the loss of part of the load. The attacker's goal is to maximize the amount of load shedding. In addition, the attacker cost function and the defender cost function are additionally set, where the attacker's cost is determined by the attacker's behavior β, and similarly, the defender's cost is determined by the defender's behavior γ.

[0021] Therefore, the attacker’s payoff function is:

[0022] Q A (β,γ)=Loss exp (β,γ)-Cost attack (β)+Cost defense (γ)

[0023] Since it is a zero-sum game, the defender’s payoff function is:

[0024] Q D (β,γ)=-Loss exp(β,γ)+Cost attack (β)-Cost defense (γ)

[0025] S2.3 Two-person zero-sum game modeling

[0026] A two-player zero-sum game is modeled. In the game, the attacker's goal is to maximize his net benefit, while the defender's goal is to minimize his total loss. The decision variables of the attacker and defender are the malware propagation rate β and the repair rate γ on each bus node, respectively. In the real world, the values ​​of these two rates depend on the attack resources and defense resources, and how they allocate resources among bus nodes.

[0027] Based on the proposed false data malware propagation and attack model, a two-person zero-sum game model is conducted for FDIA attackers and defenders in smart grids, which is defined as the following six-tuple:

[0028] <N player ,X,A,D,Q A ,Q D >

[0029] Among them, N player ={Att,Def} represents the FDIA attacker and defender agents participating in the game. In the real world, there may be more than one attacker and defender, but in order to simplify unnecessary assumptions, all attackers and all defenders are regarded as a whole in the present invention. Therefore, they are simplified to one attacker and one defender. X = (x 1 ,…,x N ) represents the state of the bus node network in the smart grid, and the binary variable x i Represents the state of the ith busbar node, where x i = 0 and 1 represent the normal state and infection state of the bus node respectively. N is the total number of bus nodes in the smart grid. A and D represent the operation sets of the attacker and defender respectively. Assume that the attacker's attack operation is β = [β 1 ,...,β N ], where β i represents the malware propagation rate on the i-th bus node, and the defender’s defense operation is γ = [γ 1 ,...,γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i All satisfy γ min ≤γ i ≤γ max ,β min ≤β i ≤β max, that is, the propagation rate and the repair rate should be between the upper and lower bounds. A and Q D Represent the reward functions of the attacker and defender defined in the previous section respectively.

[0030] Step 3: Build interactive scenes and task spaces

[0031] S3.1 Definition of state space and action space

[0032] In order to build the interactive scene and task space for the intelligent agent interaction, we first need to define the state space and action space at the code level. Based on the modeling of the two-person zero-sum game in the previous step, the state space is defined as an N-dimensional binary variable X = (x 1 ,…,x N ), define the action space as two continuous variables β = [β 1 ,…,β N ] and γ=[γ 1 ,…,γ N ], since the propagation rate and repair rate expressed by it are mathematical probabilities, the action space should satisfy 0≤γ min ≤γ i ≤γ max <1,0≤β min ≤β i ≤β max <1. In general, each agent on both sides of the attack and defense has its own independent action space, but shares the same state space, which is also in line with the attack and defense scenarios in smart grids.

[0033] The present invention uses several common space types to declare and define the state space and task space. 1 ,…,x N ), defined using multidimensional binary variables, where the value of each dimension can only be 0 or 1, indicating whether a bus node in the smart grid has been infected. For two continuous variables β = [β 1 ,...,β N ] and γ=[γ 1 ,…,γ N ], first use multidimensional continuous variables to define them separately, which defines a continuous multidimensional space within a given range, and then use dictionary type variables to combine the two subspaces into a dictionary structure. The action space here consists of two subspaces, one representing the "Attacker" and the other representing the "Defender".

[0034] S3.2 implements simulation flow calculation function

[0035] When setting the reward function of the attacker and defender in the modeling stage, the effect of attack destruction Loss (β, γ) is an important component, which is defined as the load shedding in the smart grid in this invention. Therefore, in order to quantitatively calculate the load loss caused before and after the attack, an interface Load (X) needs to be implemented when building the interactive scene, which is used to calculate the current load situation for a certain state X of the smart grid.

[0036] In this step, the power system model is first created and the parameters of each node are initialized, including the active load PD = (pd 1 ,...,pd N ), reactive load QD = (qd 1 ,...,qd N ), voltage amplitude VM=(vm 1 ,...,vm N ) etc. Then, according to the current state X, the active load value pd of the infected node i in the power system model is i Set to 0, which simulates the situation where the node load is attacked or fails. Finally, perform the optimal power flow calculation in the DC power flow calculation mode and return the result load X .

[0037] S3.3 Building interactive scenes

[0038] With the foundation of the first two steps, the interactive scene needs to implement the state transfer function

[0039] Trans(X,β,γ) is used to output the new state X′ and the reward function Q of the attacker and defender given the current state X and the actions β and γ of the attacker and defender. A and Q D .

[0040] The logic of the state transfer part is based on the SIS model. For each node i, if its state variable x i =1, that is, the node has been infected by malware, then a random function is used to randomly generate a random number rand between 0 and 1 i , if there is rand i <γ i , then the node is successfully repaired and x is obtained i ′=0; if its state variable x i = 0, that is, the node has not been infected by malware, then for each node that satisfies α ij =1 and x j = 1, and generate random numbers rand between 0 and 1 respectively. i,j , if there is rand i,j <β i , then the node is successfully infected, and we get xi ′=1, otherwise x i ′=0.

[0041] For the reward function, define:

[0042] Q A = Load(X′)-Load(X)-Cost attack (β)+Cost defend (γ), where Cost attack (β) and

[0043] Cost defend (γ) is the attack and defense cost of both sides, which can be defined as and Among them, k 1 and k 2 is the cost parameter for attack and defense. Due to the zero-sum nature of the game, there is also Q B =-Q A .

[0044] Based on the above definition, the interactive environment is constructed as Env = {X, Trans (X, β, γ)}.

[0045] Step 4: Solve the optimal defense strategy based on the deep deterministic policy gradient algorithm

[0046] S4.1 Defining DDPG Agents

[0047] Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows:

[0048] First, define the network structure and parameter update method of the DDPG agent. The Buffer class is responsible for storing the experience of the agent's interaction with the environment, storing the old state, new state, action, and reward function in the form of a tuple {X, X′, β, Q)}. The capacity of the Buffer class is fixed. If the capacity is exhausted when new experience is pushed to the Buffer, the oldest experience will be overwritten.

[0049] Actor is a feedforward neural network that receives the current state X and outputs an action β. The network contains three hidden layers, each with a different number of neurons, and the final output action is scaled to [β min ,β max ] or [γ min ,γ max ].

[0050] Critic is also a feedforward neural network that receives the current state X and action β and outputs the corresponding Q value, which is the value of action β under the current state X. Critic is similar to Actor and also has three hidden layers, but the input is the concatenation of state and action.

[0051] The DDPG agent contains a Buffer class, an Actor network and a Critic network, as well as their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process.

[0052] In addition, the DDPG agent provides a parameter update method Update() for the Actor and Critic networks. This method is responsible for updating the Actor and Critic networks, and performing soft updates on the target networks Actor_t and Critic_t.

[0053] S4.2 Initialize and train the attack and defense agent

[0054] Before starting training, first define the environment Env and the two agents Attacker and Defender, representing the attacker and defender respectively. When using the DDPG algorithm to solve the two-player zero-sum game problem, the key is to ensure that both parties converge to the optimal solution. The DDPG algorithm itself is designed to solve the reinforcement learning problem in the continuous action space, but when dealing with two-player zero-sum games, some specific adjustments and optimization strategies are required in the training method. In order to enable both parties to obtain their own better solution, the present invention chooses a training strategy of alternating training of the attacker and the defender.

[0055] Specifically, the steps to train the attacker are as follows:

[0056] The first step is to randomly initialize the environment state. For each bus node in the smart grid, randomly determine whether it is in an infected state, where X random is a randomly generated N-dimensional binary variable that represents the initial state of the environment.

[0057] Env.X=X random

[0058] In the second step, the attacker and defender choose their own actions. In order to introduce randomness in strategy exploration and avoid falling into local optimality, the actions selected by the attacker should be supplemented with additional noise ξ.

[0059] β=Attacker.Actor(X)+ξ

[0060] γ=Defender.Actor(X)

[0061] The third step is to obtain environmental feedback. The environment Env outputs the new state X′ and the reward function Q of the attacker and defender based on the current state X and the actions β and γ of the attacker and defender agents. A and Q D .

[0062] X′,Q A ,Q B =Env.Trans(X,β,γ)

[0063] Step 4: Update the attacker. A} is stored in the Attacker's Buffer as one of the samples for future training. Call Attacker's update function Update() to iterate its Actor and Critic network parameters.

[0064] Attacker.update()

[0065] Step 5: Repeat steps 1 to 4 10 times to complete this round of training for the attacker.

[0066] The steps for training the defender are similar, except that noise ξ is added to the defender action γ when selecting actions, and the parameters of the Defender's Actor and Critic networks are iterated when updating parameters. The training of the Attacker and Defender is performed alternately for multiple rounds until the loss functions of both sides converge.

[0067] The training process of each round of Attacker and Defender is as follows Figure 1 As shown. The upper part of the figure represents the two intelligent agents, Attacker and Defender, which include their respective network parameter update methods Update(), experience pool Buffer, Actor network, Critic network and corresponding target networks Actor_t and Critic_t. The lower part of the figure represents the interactive environment Env, the main part of which is the created power system model. The figure takes the IEEE14 bus node system as an example, where each number represents a bus node, and there are generators at nodes 1, 2, 3, 6, and 8. The arrow represents that the active power of the node is not zero.

[0068] S4.3 Solving the Optimal Defense Strategy

[0069] After the training of Attacker and Defender is completed, both the attacking and defending agents can output the corresponding attack strategy β and defense strategy γ when given a power network state X, so as to maximize their own reward function. Therefore, it is only necessary to input the current state X into the Actor network of Defender to obtain the approximate optimal defense strategy γ * .

[0070] γ * =Defender.Actor(X)

[0071] The advantages of the present invention compared with the prior art are:

[0072] (1) The present invention models malware propagation as a continuous process based on the SIS model, which can more accurately capture the state changes of malware propagation at any time. This means that finer time granularity and higher accuracy can be obtained without being restricted by a fixed time step. This modeling method can more accurately describe and analyze the propagation dynamics of malware, especially in scenarios that require fine time granularity and complex dynamic behavior analysis, which has obvious advantages.

[0073] (2) The present invention introduces a deep deterministic policy gradient algorithm, which combines deep learning and policy gradient methods to effectively optimize policies in high-dimensional, continuous action spaces. The Q value of the action is directly output through the policy network, without the need to discretize the continuous action space or perform complex action sampling, thus solving the dimensionality explosion problem caused by the infinite action space. At the same time, using deterministic policies instead of random policies for updating can more efficiently optimize policies in high-dimensional continuous action spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 Schematic diagram of feature grayscale image.

[0075] The symbols in the figure are explained as follows:

[0076] X represents the current environmental status;

[0077] β and γ represent the actions of Attacker and Defender, respectively;

[0078] X′ represents the new state of the environment after the attacker and defender perform their actions;

[0079] Q A and Q B Represent the reward functions of the attacker and the defender respectively. DETAILED DESCRIPTION

[0080] The method for defending against false data injection attacks on smart grids based on a two-person zero-sum game proposed in the present invention needs to solve the following three problems: (1) how to model malware propagation as a continuous process; (2) how to set the reward function for both the attacker and the defender; and (3) how to apply a deep deterministic policy gradient algorithm to make it more suitable for the present invention.

[0081] The main implementation idea of ​​the present invention is: the present invention models the false data injection attack and defense behavior in the smart grid based on the two-person zero-sum game theory, sets the reward function, and solves the approximate Nash equilibrium point in the game through the deep deterministic policy gradient algorithm, thereby obtaining the best defense strategy. First, the propagation process of the false data injection attack malware is modeled based on the SIS propagation model, and the time granularity is further refined and the accuracy is improved. Then, based on the two-person zero-sum game theory, the reward function of the attacker and the defender is decomposed into three parts: attack effect, attack cost and defense cost, making full use of the zero-sum nature in the game. Then, the Matpower simulation platform and OpenAI Gym are used to build interactive scenes and task spaces, and real-time flow calculations can be performed on any state of the power network to determine the effectiveness of the attack. Finally, the DDPG algorithm is used to solve the dimensional explosion problem caused by the infinite action space, and the strategy optimization is performed efficiently and stably to solve the best defense strategy.

[0082] The present invention proposes a method for defending against false data injection attacks in smart grids based on a two-person zero-sum game. The specific implementation steps of the present invention are described below:

[0083] Step 1: Construct a smart grid network structure model

[0084] First, the network topology of the smart grid is modeled. Assuming that the network has N bus nodes, its network topology is modeled as G =<V,E> , where V is the set of bus nodes (|V|=N), and E is the set of communication channels between them. The adjacency matrix can represent the connection relationship of nodes in the graph. Let M=(α ij ) N×N represents the adjacency matrix of G, where if there is a communication channel between the i-th and j-th bus nodes, then α ij =1, otherwise α ij =0.

[0085] Taking the 14-bus node system given by the Institute of Electrical and Electronics Engineers (IEEE) as an example, the number of bus nodes N is 14, and the adjacency matrix M = (α ij ) 14×14 The values ​​are as follows:

[0086]

[0087] Step 2: Build a false data injection attack and defense model

[0088] S2.1 Modeling of malware propagation process

[0089] After establishing the network topology model of the smart grid, the propagation process of the false data injection attack malware is modeled on its basis. In any time slot, each bus node is in one of two states: "normal state" and "infected state". The former indicates that the bus node is not infected with false data malware in this time slot, and the latter indicates that it is infected with false data malware. Using the binary variable x i,t To represent the state of the i-th bus node in the t-th time slot. i,t = 0 and 1 represent the normal state and infection state of the bus respectively. On this basis, use the vector X t =(x i,t ,...,x N,t ) is used to represent the state of the bus node network in the tth time period. Let S i,t and I i,t They represent the probability that the i-th busbar node is in a normal state and an infected state in the t-th time slot. i,t and I i,t is related (i.e. S i,t +I i,t =1), so the overall communication network state of the smart grid in the tth time slot is expressed as I t =(I i,t ,...,I N,t ).

[0090] In information systems, malware can usually be removed by deploying antivirus programs. Therefore, the state of the bus node is affected by both the propagation rate of fake data malware and the repair rate of antivirus programs. A bus node i with a normal state may be j The probability of being infected by fake data malware from the adjacent, infected bus node j is γ. Similarly, the infected bus node i can be restored to normal by the antivirus program with a repair rate of γ i .

[0091] The transition of the busbar node between these two states is represented by a differential system. Let ε represent a very small time interval, and the probability of any normal busbar node being infected with fake data malware at time t+ε is estimated as:

[0092]

[0093] According to the continuous-time Markov chain theory, this probability estimate becomes exact when ε→0. Therefore, this formula accurately estimates the probability of a normal bus node being infected with fake data malware. The estimation model first calculates the probability that a normal bus node is infected with fake data malware from a neighboring infected bus node. Based on this, the exact probability of a normal bus node being infected with fake data malware is calculated by taking the sum of these probabilities.

[0094] Similarly, the probability that any infected bus node will have its fake data malware deleted in the next time period t+ε is estimated as:

[0095] Pr{x i,t+ε =0|x i,t =1}=εγ j

[0096] Therefore, the estimated probability that any normal busbar node maintains its normal state and the probability that any infected busbar node maintains its infected state is:

[0097]

[0098] Pr{x i,t+ε =1x i,t =1}=1-εγ j

[0099] Combining all the above formulas, we can get the total probability formula:

[0100]

[0101] This formula can also be written as:

[0102]

[0103] Then let ε→0, and the following differential system can be used to describe the state evolution of the bus node network in the smart grid:

[0104]

[0105] Among them, I 1 is the initial state of the system, and T is the total number of time slots.

[0106] S2.2 Setting the reward function for both attackers and defenders

[0107] In order to establish a game model, it is also necessary to set a reward function for both the attacker and the defender. Assume that Q A and Q D Represent the reward functions of the attacker and the defender respectively. Due to the zero-sum property in the two-player zero-sum game, we can get Q A (β,γ)+Q D(β,γ)=0 is always true. This means that in the game process, the expected benefit of the attacker is also the expected loss of the defender, which is the expected damage effect of the attack. exp (β,γ), the cost of the attacker's attack attack (β) and the defender's defense cost Cost defense The destructive effect of the attack is defined as the amount of load shedding in the smart grid. When the false data malware successfully attacks a node, the node will return an erroneous measurement result to the control center, causing the control center to misjudge the situation of the entire system and re-analyze the power flow, resulting in the loss of part of the load. The attacker's goal is to maximize the amount of load shedding. In addition, the attacker cost function and the defender cost function are additionally set, where the attacker's cost is determined by the attacker's behavior β, and similarly, the defender's cost is determined by the defender's behavior γ.

[0108] Therefore, the attacker’s payoff function is:

[0109] Q A (β,γ)=Loss exp (β,γ)-Cost attack (β)+Cost defense (γ)

[0110] Since it is a zero-sum game, the defender’s payoff function is:

[0111] Q D (β,γ)=-Loss exp (β,γ)+Cost attack (β)-Cost defense (γ)

[0112] S2.3 Two-person zero-sum game modeling

[0113] A two-player zero-sum game is modeled. In the game, the attacker's goal is to maximize his net benefit, while the defender's goal is to minimize his total loss. The decision variables of the attacker and defender are the malware propagation rate β and the repair rate γ on each bus node, respectively. In the real world, the values ​​of these two rates depend on the attack resources and defense resources, and how they allocate resources among bus nodes.

[0114] Based on the proposed false data malware propagation and attack model, a two-person zero-sum game model is conducted for FDIA attackers and defenders in smart grids, which is defined as the following six-tuple:

[0115] <N player ,X,A,D,Q A ,Q D >

[0116] Where N player ={Att,Def} represents the FDIA attacker and defender agents participating in the game. In the real world, there may be more than one attacker and defender, but in order to simplify unnecessary assumptions, all attackers and all defenders are regarded as a whole in the present invention. Therefore, they are simplified to one attacker and one defender. X = (x 1 ,...,x N ) represents the state of the bus node network in the smart grid, and the binary variable x i Represents the state of the ith busbar node, where x i = 0 and 1 represent the normal state and infection state of the bus node respectively. N is the total number of bus nodes in the smart grid. A and D represent the operation sets of the attacker and defender respectively. Assume that the attacker's attack operation is β = [β 1 ,...,β N ], where β i represents the malware propagation rate on the i-th bus node, and the defender’s defense operation is γ = [γ 1 ,...,γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i All satisfy γ min ≤γ i ≤γ max ,β min ≤β i ≤β max , that is, the propagation rate and the repair rate should be between the upper and lower bounds. A and Q D Represent the reward functions of the attacker and defender defined in the previous section respectively.

[0117] Step 3: Build interactive scenes and task spaces

[0118] S3.1 Definition of state space and action space

[0119] In order to build the interactive scene and task space for the intelligent agent interaction, we first need to define the state space and action space at the code level. Based on the modeling of the two-person zero-sum game in the previous step, the state space is defined as an N-dimensional binary variable X = (x 1 ,...,x N ), define the action space as two continuous variables β = [β 1 ,...,β N ] and γ=[γ 1 ,...,γ N], since the propagation rate and repair rate expressed by it are mathematical probabilities, the action space should satisfy 0≤γ min ≤γ i ≤γ max <1,0≤β min ≤β i ≤β max <1. In general, each agent on both sides of the attack and defense has its own independent action space, but shares the same state space, which is also in line with the attack and defense scenarios in smart grids.

[0120] OpenAI Gym is a toolkit for developing and comparing reinforcement learning algorithms. It provides a set of standard environment interfaces that allow researchers and developers to easily build, train, and evaluate reinforcement learning models. Gym is widely used in academic research, industrial applications, and the learning process of RL algorithm development. With flexible scalability, Gym allows users to customize the environment according to their needs, and even create their own problems by modifying or extending existing environments. It is compatible with various reinforcement learning libraries and deep learning frameworks, such as PyTorch, TensorFlow, etc.

[0121] This time, we use several common space types provided by OpenAI Gym to declare and define state space and task space. For a 14-dimensional binary variable X = (x 1 ,...,x 14 ), defined using spaces.MultiBinary type variables, each dimension of which can only be 0 or 1, indicating whether a bus node in the IEEE14 bus node system has been infected. For two continuous variables β = [β 1 ,...,β 14 ] and γ=[γ 1 ,...,γ 14 ], first use the spaces.Box type variable to define them separately, which defines a continuous multidimensional space within a given range, and then use the spaces.Dict type variable to combine the two subspaces into a dictionary structure. The action space here consists of two subspaces, one representing the "Attacker" and the other representing the "Defender".

[0122] S3.2 implements simulation flow calculation function

[0123] When setting the reward function for both the attacker and the defender in the modeling phase, the effect of attack damage Loss exp(β,γ) is an important component, which is defined as the load shedding amount in the smart grid in this invention. Therefore, in order to quantitatively calculate the load loss caused before and after the attack, an interface Load(X) needs to be implemented when building an interactive scenario, which is used to calculate the current load situation for a certain state X of the smart grid.

[0124] In this step, firstly, the parameter 'case14' is passed to the loadcase() function provided by the simulation tool Matpower to create the IEEE14 bus node power system model and initialize the parameters of each node, including the active load PD = (pd 1 ,...,pd 14 ), reactive load QD = (qd 1 ,...,qd 14 ), voltage amplitude VM=(vm 1 ,...,vm 14 ) etc. Then, according to the current state X, the active load value pd of the infected node i in the power system model is i Set to 0, which simulates the situation where the node load is attacked or fails. Finally, configure it to DC power flow calculation mode through mpoption(mpoption,'PF_DC',1), and call runopf() function to run the optimal power flow calculation and return the result load X =Load(X)=Loss exp (β,γ).

[0125] S3.3 Building interactive scenes

[0126] With the foundation of the first two steps, the interactive scene needs to implement the state transfer function

[0127] Trans(X,β,γ) is used to output the new state X′ and the reward function Q of the attacker and defender given the current state X and the actions β and γ of the attacker and defender. A and Q D .

[0128] The logic of the state transfer part is based on the SIS model. For each node i, if its state variable x i =1, that is, the node has been infected by malware, then use random.random() to randomly generate a 0-1 random number rand i , if there is rand i <γ i , then the node is successfully repaired and x is obtained i ′=0; if its state variable x i = 0, that is, the node has not been infected by malware, then for each node that satisfies α ij=1 and x j = 1, and generate random numbers rand between 0 and 1 respectively. i,j , if there is rand i,j <β i , then the node is successfully infected, and we get x i ′=1, otherwise x i ′=0.

[0129] For the reward function, define Q A = Load(X′)-Load(X)-Cost attack (β)+Cost defend (γ), where Cost attack (β) and Cost defend (γ) is the attack and defense cost of both sides, which can be defined as and where k 1 and k 2 is the cost parameter for attack and defense. Due to the zero-sum nature of the game, there is also Q B =-Q A .

[0130] Based on the above definition, the interactive environment is constructed as Env = {X, Trans (X, β, γ)}.

[0131] Step 4: Solve the optimal defense strategy based on the deep deterministic policy gradient algorithm

[0132] S4.1 Defining DDPG Agents

[0133] Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows:

[0134] First, define the network structure and parameter update method of the DDPG agent. The Buffer class is responsible for storing the experience of the agent's interaction with the environment, storing the old state, new state, action, and reward function in the form of a tuple {X, X′, β, Q)}. The capacity of the Buffer class is fixed. If the capacity is exhausted when new experience is pushed to the Buffer, the oldest experience will be overwritten.

[0135] Actor is a feedforward neural network that receives the current state X and outputs an action β. The network contains three hidden layers, each with a different number of neurons, and the final output action is scaled to [β min ,β max ] or [γ min ,γ max ].

[0136] Critic is also a feedforward neural network that receives the current state X and action β and outputs the corresponding Q value, which is the value of action β under the current state X. Critic is similar to Actor and also has three hidden layers, but the input is the concatenation of state and action.

[0137] The DDPG agent contains a Buffer class, an Actor network and a Critic network, as well as their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process.

[0138] In addition, the DDPG agent provides a parameter update method Update() for the Actor and Critic networks. This method is responsible for training the Actor and Critic networks and mainly includes the following steps:

[0139] In the first step, a data sample {X, X′, β, Q} is randomly sampled from the Buffer class. This sample is used to calculate the network's loss function and perform gradient descent.

[0140] The second step is to calculate the target Q value Target_Q. Target_Q is calculated by the target network Actor_t and Critic_t to guide the learning of the Critic network. Specifically, the action β or γ corresponding to the next state is first calculated by the Actor target network Actor_t, and then the Q value Q of the action is calculated by the Critic target network Critic_t. future , as shown below.

[0141] Q future =Critic_t(X′,Actor_t(X′))

[0142] The target Q value Target_Q represents the discounted sum of current rewards and future rewards, as shown in the following formula, where α is the discount factor used to balance the importance of current and future rewards.

[0143] Target_Q=Q+αQ future

[0144] The third step is to calculate the current Q value Current_Q. Current_Q is calculated by the current Critic network, which represents the value of the agent performing the current action β in the current state X.

[0145] Current_Q=Critic(X,β)

[0146] The fourth step is to optimize the Critic network. The loss function of the Critic network is the mean square error between the current Q value Current_Q and the target Q value Target_Q. The smaller the loss function, the more accurate the Critic network's estimate of the Q value.

[0147] After calculating the loss, the back-propagation algorithm is used to update the parameters of the Critic network.

[0148] The fifth step is to optimize the Actor network. The loss function of the Actor network is defined as the negative of the Q value calculated by the Critic network. This means that the goal of the Actor network is to find an action that maximizes the Q value output by the Critic network, thereby selecting the optimal strategy.

[0149] After calculating the loss, the back-propagation algorithm is used to update the parameters of the Actor network.

[0150] Step 6: Update the target network Actor_t and Critic_t. The parameters of Actor_t and Critic_t are updated using soft update. Specifically, the parameters of Actor_t and Critic_t are the weighted sum of the current Actor and Critic network parameters and their own parameters. The purpose of soft update is to make the parameters of the target network change more smoothly and stably, thereby improving the stability of the training process.

[0151] S4.2 Initialize and train the attack and defense agent

[0152] Before starting training, first define the environment Env and the two agents Attacker and Defender, representing the attacker and defender respectively. When using the DDPG algorithm to solve the two-player zero-sum game problem, the key is to ensure that both parties converge to the optimal solution. The DDPG algorithm itself is designed to solve the reinforcement learning problem in the continuous action space, but when dealing with two-player zero-sum games, some specific adjustments and optimization strategies are required in the training method. In order to enable both parties to obtain their own better solution, the present invention chooses a training strategy of alternating training of the attacker and the defender.

[0153] Specifically, the steps to train the attacker are as follows:

[0154] In the first step, the environment state is randomly initialized. For each bus node in the smart grid, it is randomly determined whether it is in an infected state.

[0155] Env.X=X random

[0156] In the second step, the attacker and defender choose their own actions. In order to introduce randomness in strategy exploration and avoid falling into local optimality, the actions selected by the attacker should be supplemented with additional noise ξ.

[0157] β=Attacker.Actor(X)+ξ

[0158] γ=Defender.Actor(X)

[0159] The third step is to obtain environmental feedback. The environment Env outputs the new state X′ and the reward function Q of the attacker and defender based on the current state X and the actions β and γ of the attacker and defender agents. A and Q D .

[0160] X′,Q A ,Q B =Env.Trans(X,β,γ)

[0161] Step 4: Update the attacker. A} is stored in the Attacker's Buffer as one of the samples for future training. Call Attacker's update function Update() to iterate its Actor and Critic network parameters.

[0162] Attacker.update()

[0163] Step 5: Repeat steps 1 to 4 10 times to complete this round of training for the attacker.

[0164] The steps for training the defender are similar, except that noise ξ is added to the defender action γ when selecting actions, and the parameters of the Defender's Actor and Critic networks are iterated when updating parameters. The training of the Attacker and Defender is performed alternately for multiple rounds until the loss functions of both sides converge.

[0165] The training process of each round of Attacker and Defender is as follows Figure 1As shown. The upper part of the figure represents the two intelligent agents, Attacker and Defender, which include their respective network parameter update methods Update(), experience pool Buffer, Actor network, Critic network and corresponding target networks Actor_t and Critic_t. The lower part of the figure represents the interactive environment Env, the main part of which is the created power system model. The figure takes the IEEE14 bus node system as an example. Each number represents a bus node, where the arrow represents the active power output of the node, and the letter G represents the presence of generators at nodes 1, 2, 3, 6, and 8.

[0166] In this step, the present invention adopts the strategy of alternating training because in a two-player zero-sum game, the strategies of both parties are changing. Because the goal of each party is to maximize its own reward and minimize the opponent's reward, when one party updates its strategy, the other party will be directly affected. Alternating training ensures that both parties can adapt to the changes of the opponent during the process of strategy training.

[0167] If you only train one side and ignore the other, the trained strategy may only be applicable to the current opponent's strategy. Once the opponent's strategy changes, the trained strategy may become invalid. Therefore, alternating training can help both sides constantly adapt to the opponent's strategy changes, thereby finding a more robust solution.

[0168] A core feature of a two-player zero-sum game is adversarial nature. When one party adopts a certain strategy, the other party's goal is to find the best response strategy to minimize the opponent's gain. In this adversarial environment, alternating training can be seen as part of adversarial learning.

[0169] During the alternating training process, the strategies of both sides are constantly improving. By constantly responding to the opponent's optimal strategy, the performance of both sides will gradually tend to be optimal. The adversarial learning mechanism enables the intelligent agent to perform well even in the worst case, which is difficult to achieve in simple fixed training.

[0170] S4.3 Solving the Optimal Defense Strategy

[0171] After the training of Attacker and Defender is completed, both agents can output the corresponding attack strategy β and defense strategy γ when given a power network state X, so as to maximize their own reward function. Therefore, it is only necessary to input the current state X into the Actor network of Defender to obtain the approximate optimal defense strategy γ. * .

[0172] γ * =Defender.Actor(X)

[0173] In order to verify the effectiveness and strategy of the Defender agent trained in this study against the false data injection attack in the IEEE14 bus node system, * The approximate optimality can be done as follows:

[0174] Assume that the IEEE14 bus node system state is fixed X = (0,1,0,0,0,0,0,0,0,0,0,0,0,0), and the attacker’s attack strategy β = β * Randomly select the defender's defense strategy 999 timesγ 1 ,γ 2 ,...,γ 999 , respectively calculate the reward function Q of these random defense strategies and the approximate optimal defense strategy when the system state and attack strategy remain unchanged 1 ,Q 2 ,...,Q 999 ,Q * , and sorted in descending order according to the size of the reward function, the results are shown below.

[0175] Serial number Defensive Strategy Q Value 1 (0.00,0.38,0.00,0.00,0.00,0.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00) -85.31 2 (0.24,0.41,0.01,0.62,0.33,0.05,0.15,0.12,0.48,0.56,0.30,0.26,0.18,0.04) -115.26 3 (0.44,0.07,0.40,0.27,0.30,0.06,0.18,0.30,0.58,0.20,0.19,0.15,0.04,0.20) -115.54 4 (0.24,0.13,0.09,0.47,0.09,0.11,0.13,0.18,0.63,0.33,0.14,0.49,0.17,0.48) -120.12 … … … 999 (0.89,0.80,0.86,0.88,0.83,0.88,0.32,0.82,0.40,0.42,0.03,0.69,0.79,0.02) -326.73 1000 (0.12,0.90,0.85,0.61,0.28,0.24,0.79,0.84,0.86,0.86,0.89,0.67,0.87,0.68) -359.74

[0176] In the table above, the defense strategy given in the first row is the approximate optimal defense strategy γ calculated by the Defender agent. * It can be seen that among the 1000 random defense strategies, the reward function value corresponding to this strategy is the largest, indicating that this strategy is optimal compared to other defense strategies. This also proves the effectiveness of the Defender agent trained in this study in defending against false data injection attacks in the IEEE14 bus node system.

[0177] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

[0178] The above is only a preferred implementation of the method for defending against false data injection attacks on smart grids based on a two-person zero-sum game of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the method for defending against false data injection attacks on smart grids based on a two-person zero-sum game of the present invention. These improvements and modifications should also be regarded as the protection scope of the method for defending against false data injection attacks on smart grids based on a two-person zero-sum game of the present invention.

Claims

1. A method for defending against false data injection attacks in smart grids based on two-person zero-sum game, characterized in that: The following steps are involved: Step 1: Construct a smart grid network structure model; Step 2: Construct a false data injection attack and defense model, including: modeling the malware propagation process, setting the reward function for both the attacker and the defender, and modeling a two-person zero-sum game. Step 3: Build interactive scenes and task spaces; including: defining state space and action space, realizing simulation flow calculation functions, and building interactive scenes; Step 4: Solve the optimal defense strategy based on the deep deterministic policy gradient algorithm; including: defining the DDPG agent, initializing and training the attack and defense agents, and solving the optimal defense strategy.

2. According to claim 1, a method for defending against false data injection attacks on smart grids based on two-person zero-sum game, characterized in that: In step 1, the network topology structure of the smart grid is modeled. Assuming that the network has N bus nodes, its network topology structure is modeled as G =<V,E> , where V is the set of bus nodes (|V| = N), and E is the set of communication channels between them; the adjacency matrix represents the connection relationship between nodes in the graph, let M = (α ij ) N×N Represents the adjacency matrix of G. If there is a communication channel between the i-th and j-th bus nodes, then α ij =1, otherwise α ij =0.

3. A method for defending against false data injection attacks on smart grids based on two-person zero-sum game according to claim 1 or 2, characterized in that: In step 2, the malware propagation process is modeled: After establishing the network topology model of the smart grid, the propagation process of the false data injection attack malware is modeled; in any time slot, each bus node is in one of two states: "normal state" and "infected state", the former means that the bus node is not infected with false data malware in this time slot, and the latter means that it is infected with false data malware; using the binary variable x i,t To represent the state of the i-th bus node in the t-th time slot; x i,t = 0 and 1 represent the normal state and infection state of the bus respectively; use vector X t =(x i,t ,...,x N,t ) to represent the state of the bus node network in the tth time slot; let S i,t and I i,t They represent the probability that the i-th busbar node is in the normal state and the infected state in the t-th time interval respectively; since S i,t and I i,t is related, that is, S i,t +I i,t =1, so the overall communication network state of the smart grid at the tth time slot is expressed as I t =(I i,t ,...,I N,t ); A busbar node i with a normal state may be j The probability of infection from the fake data malware of the adjacent, infected bus node j is γ; similarly, the infected bus node i is restored to normal state by the antivirus program, and the repair rate is γ i ; The transition of a busbar node between two states is represented by a differential system: Among them, I1 is the initial state of the system, and T is the total number of time slots.

4. The method for defending against false data injection attacks of smart grid based on two-person zero-sum game according to claim 3 is characterized in that: In step 2, set the reward function for both attacker and defender: In order to establish a game model, it is also necessary to set a reward function for both the attacker and the defender. Let Q A and Q D Represent the reward functions of the attacker and the defender respectively. Due to the zero-sum property in the two-player zero-sum game, we get Q A (β,γ)+Q D (β,γ)=0 is always true; in the game process, the expected benefit of the attacker is also the expected loss of the defender, which is the expected damage effect of the attack. exp (β,γ), the cost of the attacker's attack Cost attack (β) and the defender's defense cost Cost defense (γ) consists of three parts; the destructive effect of the attack is defined as the amount of load shedding in the smart grid; when the false data malware successfully attacks a node, the node will return an erroneous measurement result to the control center, causing the control center to misjudge the situation of the entire system and re-analyze the power flow, resulting in the loss of part of the load; the attacker's goal is to maximize the amount of load shedding. In addition, the attacker cost function and the defender cost function are additionally set, where the attacker's cost is determined by the attacker's behavior β, and similarly, the defender's cost is determined by the defender's behavior γ; Therefore, the attacker’s payoff function is: Q A (b,c)=Loss exp (b,c)-Cost attack (b)+Cost defense (c) Since it is a zero-sum game, the defender’s payoff function is: Q D (b,c)=-Loss exp (b,c)+Cost attack (b)-Cost defense (c)。 5. A method for defending against false data injection attacks on smart grids based on two-person zero-sum game according to claim 1 or 4, characterized in that: In step 2, the two-player zero-sum game is modeled: A two-player zero-sum game is modeled; in the game, the attacker's goal is to maximize his net benefit, while the defender's goal is to minimize his total loss; the decision variables of the attacker and defender are the malware propagation rate β and the repair rate γ on each bus node, respectively; the values ​​of these two rates depend on the attack resources and defense resources, and how they allocate resources among the bus nodes; Based on the proposed false data malware propagation and attack model, a two-person zero-sum game model is conducted for FDIA attackers and defenders in smart grids, which is defined as the following six-tuple: <N player ,X,A,D,Q A ,Q D > Among them, N player = {Att, Def} represents the FDIA attacker and defender agents participating in the game; there may be more than one attacker and defender, and all attackers and defenders are defined as a whole; therefore, it is simplified to one attacker and one defender; X = (x1, ..., x N ) represents the state of the bus node network in the smart grid, and the binary variable x i Represents the state of the ith busbar node, where x i = 0 and 1 represent the normal state and infection state of the bus node respectively; N is the total number of bus nodes in the smart grid; A and D represent the operation sets of the attacker and defender respectively. Suppose the attack operation of the attacker is β = [β1,...,β N ], where β i represents the malware propagation rate on the i-th bus node, and the defender’s defense operation is γ=[γ1,...,γ N ], where γ i represents the repair rate of malware on the i-th bus node, where β i and γ i All meet the following requirements: γ min ≤γ i ≤γ max ,β min ≤β i ≤β max , that is, the propagation rate and the repair rate should be between the upper and lower bounds; Q A and Q D Represent the reward functions of the attacker and defender defined in the previous section respectively.

6. The method for defending against false data injection attacks of smart grid based on two-person zero-sum game according to claim 1, characterized in that: In step three, define the state space and action space: Define the state space as N-dimensional binary variables X = (x1,...,x N ), define the action space as two continuous variables β=[β1,...,β N ] and γ=[γ1,...,γ N ], since the propagation rate and repair rate expressed by it are mathematical probabilities, the action space should satisfy 0≤γ min ≤γ i ≤γ max <1,0≤β min ≤β i ≤β max <1; Each agent on both sides has its own independent action space, but shares the same state space, which is consistent with the attack and defense scenarios in smart grids; For an N-dimensional binary variable X=(x1,...,x N ), using a multidimensional binary variable definition, the value of each dimension can only be 0 or 1, indicating whether a bus node in the smart grid has been infected; for two continuous variables β = [β1, ..., β N ] and γ=[γ1,...,γ N ], first use multidimensional continuous variables to define them separately, which defines a continuous multidimensional space within a given range, and then use dictionary type variables to combine the two subspaces into a dictionary structure; the action space here is composed of two subspaces, one representing the "attacker" and the other representing the "defender".

7. A method for defending against false data injection attacks on smart grids based on two-person zero-sum game according to claim 1 or 6, characterized in that: In step three, the simulation flow calculation function is realized: When setting the reward function for both the attacker and the defender in the modeling phase, the effect of the attack damage Loss (β, γ) is an important component, which is defined as the load shedding amount in the smart grid. In order to quantitatively calculate the load loss caused before and after the attack, an interface Load (X) needs to be implemented when building the interactive scenario. Its function is to calculate the current load situation for a certain state X of the smart grid. First, create a power system model and initialize the parameters of each node, including active load PD = (pd1, ..., pd N ), reactive load QD=(qd1,...,qd N ), voltage amplitude VM=(vm1,...,vm N ); Then, according to the current state X, the active load value pd of the infected node i in the power system model is i Set to 0, and finally perform the optimal power flow calculation in the DC power flow calculation mode, and return the result load X .

8. The method for defending against false data injection attacks of smart grid based on two-person zero-sum game according to claim 7, characterized in that: In step three, build the interactive scene: The interactive scenario requires the implementation of the state transfer function Trans(X,β,γ), which outputs the new state X′ and the reward function Q of the attacker and defender given the current state X and the actions β and γ of the attacker and defender. A and Q D ; The logic of the state transfer part is based on the SIS model. For each node i, if its state variable x i =1, that is, the node has been infected by malware, then a random function is used to randomly generate a random number rand between 0 and 1 i , if there is rand i <γ i , then the node is successfully repaired and x is obtained i ′=0; If its state variable x i = 0, that is, the node has not been infected by malware, then for each node that satisfies α ij =1 and x j = 1, and generate random numbers rand between 0 and 1 respectively. i,j , if there is rand i,j <β i , then the node is successfully infected, and we get x i ′=1, otherwise x i ′=0; For the reward function, define: Q A =Load(X′)-Load(X)-Cost attack (β)+Cost defend (γ), Among them, Cost attack (β) and Cost defend (γ) is the attack and defense cost of both sides, which is defined as and Among them, k1 and k2 are the cost parameters of attack and defense; due to the zero-sum nature of the game, there is also Q B =-Q A ; Build the interactive environment Env = {X, Trans (X, β, γ)}.

9. The method for defending against false data injection attacks in smart grid based on two-person zero-sum game according to claim 1, characterized in that: In step 4, define the DDPG agent: Define {Buffer, Actor, Critic, Actor_t, Critic_t, update()} as the DDPG agent structure, where each part is defined as follows: First, define the network structure and parameter update method of the DDPG agent; the Buffer class is responsible for storing the experience of the agent's interaction with the environment, storing the old state, new state, action, and reward function in the form of a tuple {X, X′, β, Q)}; the capacity of the Buffer class is fixed. If the capacity is exhausted when new experience is pushed to the Buffer, the oldest experience will be overwritten; Actor is a feedforward neural network that receives the current state X and outputs an action β; the network contains three hidden layers, each with a different number of neurons, and the final output action is scaled to [β min ,β max ] or [γ min ,γ max ]; Critic is also a feedforward neural network. It receives the current state X and action β and outputs the corresponding Q value, which is the value of action β under the current state X. Critic also has three hidden layers, but the input is the concatenation of state and action. The DDPG agent contains a Buffer class, an Actor network and a Critic network, as well as their corresponding target networks Actor_t and Critic_t, which are used to stabilize the training process; In addition, the DDPG agent provides a parameter update method Update() for the Actor and Critic networks. This method is responsible for updating the Actor and Critic networks, and performing soft updates on the target networks Actor_t and Critic_t.

10. The method for defending against false data injection attacks of smart grid based on two-person zero-sum game according to claim 9, characterized in that: In step 4, initialize and train the attack and defense agents: Before starting training, we first define the environment Env and two agents, Attacker and Defender, which represent the attacker and defender respectively. When using the DDPG algorithm to solve the two-player zero-sum game problem, the key is to ensure that both parties converge to the optimal solution. We also select a training strategy that alternately trains the attacker and defender. Specifically, the steps to train the attacker are as follows: The first step is to randomly initialize the environment state; for each bus node in the smart grid, randomly determine whether it is in an infected state, where X random is a randomly generated N-dimensional binary variable, representing the initial state of the environment; Env.X=X random In the second step, the attacker and defender choose their own actions. In order to introduce randomness into the strategy exploration and avoid falling into the local optimum, the actions selected by the attacker should be supplemented with additional noise ξ. β=Attacker.Actor(X)+ξ γ=Defender.Actor(X) The third step is to obtain environmental feedback; the environment Env outputs the new state X′ and the reward function Q of the attacker and defender based on the current state X and the actions β and γ of the attacker and defender agents. A and Q D ; X′,Q A ,Q B =Env.Trans(X,β,γ) Step 4: Update the attacker; update the interaction process {X,X′,β,Q A } Store it in the Attacker's Buffer as one of the samples for future training; call the Attacker's update function Update() to iterate its Actor and Critic network parameters; Attacker.update() Step 5: Repeat steps 1 to 5 10 times to complete this round of training for the attacker. The steps of training the defender are different from those of training the attacker in that noise ξ is added to the defender action γ when selecting actions, and the parameters of the Defender's Actor and Critic networks are iterated when updating parameters; the training of the Attacker and Defender is performed alternately for multiple rounds until the loss functions of both sides converge.

11. The method for defending against false data injection attacks of smart grid based on two-person zero-sum game according to claim 10, characterized in that: In step 4, the optimal defense strategy is solved: After the training of Attacker and Defender is completed, both the attacking and defending agents can output the corresponding attack strategy β and defense strategy γ when a power network state X is given, so as to maximize their own reward function; therefore, it is only necessary to input the current state X into the Actor network of Defender to obtain the approximate optimal defense strategy γ * ; c * =Defender.Actor(X).

Images