A mobile target defense method, device and storage medium based on reinforcement learning

Reinforcement learning-based defense strategies enhance power grid resilience against FDIA by optimizing bus switching operations, improving detection efficiency to 93% in a five-bus system, addressing the limitations of static and mobile target defense methods.

CN115694904BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211193174.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-28
Publication Date
2024-12-03
Estimated Expiration
2042-09-28

AI Technical Summary

Technical Problem

Existing power grid defense systems are vulnerable to false data injection attacks (FDIA) that can bypass bad data detection, leading to incorrect state estimation and operational harm, with static defense methods being costly and difficult to implement, and existing mobile target defense methods failing to achieve optimal detection efficiency.

Method used

A method utilizing reinforcement learning to build an attack scenario library, perform bus switching operations, and train an agent to generate a defense strategy that maximizes detection efficiency by updating parameters based on system states and rewards, incorporating least squares state estimation and bad data monitoring.

Benefits of technology

Improves detection rate of FDIA by achieving a 93% detection rate in a five-bus system, outperforming genetic algorithms, while avoiding system disconnection and thermal limit violations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115694904B_ABST
    Figure CN115694904B_ABST
Patent Text Reader

Abstract

The present invention discloses a mobile target defense method, device and storage medium based on reinforcement learning, which relates to the technical field of power system attack defense. The method uses a designed false data injection attack to build an attack scenario library for training a reinforcement learning agent; performs a switching operation on a system bus, connects the bus, load and generator set to a spare bus to change the topology of the power system, thereby adding a defense action; evaluates the detection rate of the false data injection attack based on a state estimation model and a bad data monitoring system; trains and updates the reinforcement learning agent, and updates the parameters of the agent according to the current state and reward; repeats the above steps until the detection rate of the reinforcement learning in response to the attack scenario library converges; generates a defense strategy based on the trained reinforcement learning agent, and the operator operates the power system according to the defense strategy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of power system attack defense, and specifically relates to a mobile target defense method, device and storage medium based on reinforcement learning. Background Art

[0002] With the continuous development of science and technology, the power grid currently uses bad data detection to filter measurement errors caused by equipment failure or malicious attacks. False data injection attack (FDIA) can effectively bypass the bad data detection system of the power system, thereby directly affecting the results of state estimation, causing the control center to make wrong decisions based on wrong results, causing harm to the normal operation of the power grid.

[0003] In some static defense methods, the game between attackers and defenders is unbalanced. Attackers can use the static characteristics of static networks to perform long-term and effective attacks. Common mobile target defense methods mainly realize the uncertainty of power grid topology by changing line parameters, but the investment cost is high and the implementation is difficult in actual systems. This method uses reinforcement learning methods to formulate the optimal strategy for bus switching operations, thereby achieving maximum detection efficiency of attack vectors using limited operations. Summary of the invention

[0004] In view of the deficiencies in the prior art, the object of the present invention is to provide a mobile target defense method, device and storage medium based on reinforcement learning.

[0005] The purpose of the present invention can be achieved by the following technical solution: a mobile target defense method based on reinforcement learning, the method comprising the following steps:

[0006] Use the designed false data injection attack to build an attack scenario library for training reinforcement learning agents;

[0007] Perform switching operations on the system busbar, connect the busbar, loads, and generator sets to the standby busbar to change the topology of the power system, thereby adding defensive actions;

[0008] Build a least squares state estimation model and bad data monitoring system, and evaluate the detection rate of false data injection attacks based on the least squares state estimation model and bad data monitoring system;

[0009] Use the constructed attack scenario library to train and update the reinforcement learning agent, and update the parameters of the agent based on the current state and reward;

[0010] Repeat the above steps until the detection rate of reinforcement learning for attack scenario library converges;

[0011] Based on the trained reinforcement learning agent, a defense strategy is generated, and the operator operates the power system according to the defense strategy.

[0012] Preferably, the process of building the attack scenario library is:

[0013] a=h(x+δ)-h(x)

[0014] Where x is the state quantity, a is the attack vector added to the quantity measurement, δ is the deviation value that the attacker wants to inject into the state quantity, and h is the nonlinear mapping relationship between the system state quantity and the quantity measurement, which is composed of the system topology and line parameter information; the construction rule of the attack scenario library is to generate N groups of different attack vectors based on random δ; in this case, the system's bad data monitoring system residual is:

[0015] r a =z a –h(x a )=z+a–h(x+δ)=z–h(x)=eta

[0016] where r a is the residual, z a represents the measurement after adding the attack vector, η represents independent measurement noise, and the noise obeys Gaussian distribution. At this time, the residual does not exceed the threshold of the bad data detection system, and the attack vector can avoid the system defense measures.

[0017] Preferably, after the defensive action occurs, the attacker still designs the attack vector based on the previous topology, and the concealment will be lost under the topology formed by the specific action.

[0018] Preferably, the detection rate of the evaluation of the false data injection attack is mainly to judge the residual after the state estimation, and compare the second norm of the residual with a predetermined threshold value τ. If it is greater than the threshold, the bad data alarm is triggered, indicating that there is abnormal data in the collected measurement value; otherwise, the measurement value is normal data, and the square of the second norm of the measurement value residual conforms to the chi-square distribution with mn degrees of freedom. The threshold τ is determined by a hypothesis test with a confidence level of mn, that is, Among them, m is the number of measured values ​​and n is the number of state quantities.

[0019] The detection rate for injection attacks is the number of detected attack vectors divided by N, where N is the number of attack vectors.

[0020] Preferably, the state space required for reinforcement learning in the reinforcement learning agent includes the system measurement information z such as voltage P, bus active power P i , reactive power P j 、Line active power P ij , reactive power Q ij , thermal limit rate rhoij .

[0021] Preferably, the selection scheme of the reinforcement learning agent action is: the actionable space includes the beginning and end of each line, the generator and the load node, which are encoded by the hot encoding vector during the reinforcement learning process.

[0022] Preferably, the reinforcement learning agent reward function is set to:

[0023]

[0024] Where ρ(FDIA) represents the detection rate of false data injection attacks. If all thermal limits do not exceed 1, the reward value is the detection rate; if any line exceeds the thermal limit, the reward value is set to 0;

[0025]

[0026] The detection rate is the number of attack vectors exceeding the threshold among N attack vectors, zi represents the state when the i-th attack vector is added, ai represents the i-th attack vector, and h(x i ,top t ) represents the mapping from the system topological state quantity to the quantity measurement at time t.

[0027] Preferably, the training process of the reinforcement learning agent is as follows: the attacker generates an attack vector based on the initial system. Under the system state at time 1, the reinforcement learning agent performs action a, the system topology changes, and the reward value is calculated. At this time, the system environment also changes. According to the new system state at time 2, the system generates a new action, and this process is repeated until the reward value stabilizes. After the process ends, the total reward value is recorded, and the reinforcement learning agent is updated according to the gradient descent method.

[0028] Beneficial effects of the present invention:

[0029] The method of the present invention improves the detection rate after each action combination. In the presence of a penalty item, the detection rate is the same as the reward value, indicating that the system has not experienced a situation where the action combination is disconnected and the line exceeds the thermal limit. Although both reinforcement strategy learning and genetic algorithms can ultimately obtain strategies with higher detection rates, due to the advantage of sequential decision-making in reinforcement learning, a higher detection rate can obviously be obtained in each operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 Flowchart of the defense method for false data injection attacks based on reinforcement learning;

[0032] Figure 2 It is a schematic diagram of topological action;

[0033] Figure 3 Detection efficiency of defense strategies against FDIA. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, a false data injection attack defense method based on reinforcement learning is divided into an attack and defense scenario construction phase and a strategy solving phase.

[0036] Step 1: Construct a false data injection attack vector for subsequent training of the reinforcement learning agent. The construction method is:

[0037] a=h(x+δ)-h(x)

[0038] Where x is the state quantity, a is the attack vector added to the quantity measurement, δ is the deviation value that the attacker wants to inject into the state quantity, and h is the nonlinear mapping relationship between the system state quantity and the quantity measurement, which is composed of the system topology and line parameter information; the construction rule of the attack scenario library is to generate N groups of different attack vectors based on random δ. In this case, the system's bad data monitoring system residual is:

[0039] r a =z a –h(x a )=z+a–h(x+δ)=z–h(x)=eta

[0040] where r a is the residual, z a represents the measurement after adding the attack vector, η represents independent measurement noise, and the noise obeys Gaussian distribution. At this time, the residual does not exceed the threshold of the bad data detection system, and the attack vector can avoid the system defense measures.

[0041] Step 2: Implement defensive actions.

[0042] The defensive action relies on the operator's switching operation on the system bus to connect the bus, load, and generator set to the standby bus to change the topology of the power system;

[0043] Step 3: Evaluate the defense effect of the current action.

[0044] The bad data detection system is used to evaluate the detection effect. It mainly judges the residual after state estimation and compares the residual norm with the predetermined threshold τ. If it is greater than the threshold, the bad data alarm is triggered, indicating that there are abnormal data in the collected measurement values; otherwise, the measurement value is normal data, and the square of the residual norm of the measurement value conforms to the chi-square distribution with mn degrees of freedom. The threshold τ is determined by the hypothesis test with a confidence level of mn, that is, Among them, m is the number of measured values, n is the number of state quantities. The final attack vector detection rate is the number of detected attack vectors divided by N, where N is the number of attack vectors.

[0045] Step 4: Build and update the reinforcement learning agent;

[0046] The state space required for reinforcement learning includes the system's measurement information z, such as voltage P, bus active power P i , reactive power P j 、Line active power P ij , reactive power Q ij , thermal limit rate rho ij .

[0047] The action space includes the beginning and end of each line, the generator and load nodes, which are encoded by one-hot encoded vectors during reinforcement learning.

[0048] The reinforcement learning agent reward function is set as:

[0049]

[0050] Where ρ(FDIA) represents the detection rate of false data injection attack, and the reward value is the detection rate if all thermal limits do not exceed 1. If any line exceeds the thermal limit, the reward value is set to 0.

[0051]

[0052] The detection rate is the number of attack vectors exceeding the threshold among N attack vectors, z i Indicates the state when the i-th attack vector is added, a i represents the i-th attack vector, h(x i ,top t ) represents the mapping from the system topological state quantity to the quantity measurement at time t.

[0053] Step 5: Train the reinforcement learning agent.

[0054] The training process is as follows: the attacker generates an attack vector based on the initial system. At the system state at time 1, the reinforcement learning agent performs action a, the system topology changes, and the reward value is calculated. At this time, the system environment also changes. According to the new system state at time 2, the system generates a new action. This action is repeated until the reward value stabilizes. After the process is completed, the total reward value is recorded, and the reinforcement learning agent is updated according to the gradient descent method.

[0055] Repeat the above process until the policy feedback is stable within the range and the training ends.

[0056] Step 6: Generate a strategy.

[0057] A defense strategy is generated based on the trained intelligent agent, and the operator operates the power system according to the defense strategy.

[0058] The proposed method is tested on a five-bus system consisting of two power plants, eight transmission lines and five substations. Figure 3 The impact of each switching operation on the system detection rate is shown. It can be seen that the detection rate is improved after each action combination. In the presence of a penalty term, the detection rate and reward value are the same, indicating that the system has not experienced a disconnection due to an action combination and the line has not exceeded the thermal limit. In addition, a comparison with the heuristic method is added. In this scenario, the training results of the genetic algorithm are obtained through the same 6,000 iterations. Figure 3 The results show that after the last step, the detection rate of FDIA based on DQN method increased to 93%, and the detection rate of GA based method was 89%. Although both reinforcement strategy learning and genetic algorithm can finally obtain strategies with higher detection rates, due to the advantages of sequential decision-making of reinforcement learning, it is obvious that a higher detection rate can be obtained at each operation.

[0059] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0060] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A mobile target defense method based on reinforcement learning, It is characterized in that The method comprises the following steps: Use the designed false data injection attack to build an attack scenario library for training reinforcement learning agents; The process of building an attack scenario library is as follows: a=h(x+δ)-h(x) Where x is the state quantity, a is the attack vector added to the quantity measurement, δ is the deviation value that the attacker wants to inject into the state quantity, and h is the nonlinear mapping relationship between the system state quantity and the quantity measurement, which is composed of the system topology and line parameter information; the construction rule of the attack scenario library is to generate N groups of different attack vectors based on random δ; in this case, the system's bad data monitoring system residual is: r a =z a –h(x a )=z+a–h(x+δ)=z–h(x)=η where r a is the residual, z a represents the measurement after adding the attack vector, η represents the independent measurement noise, and the noise follows a Gaussian distribution. At this time, the residual does not exceed the threshold of the bad data detection system, and the attack vector can avoid the system defense measures; Perform switching operations on the system busbar, connect the busbar, loads, and generator sets to the standby busbar to change the topology of the power system, thereby adding defensive actions; Build a least squares state estimation model and bad data monitoring system, and evaluate the detection rate of false data injection attacks based on the least squares state estimation model and bad data monitoring system; The detection rate of false data injection attack is evaluated by judging the residual after state estimation, and comparing the residual norm with the predetermined threshold τ. If it is greater than the threshold, the bad data alarm is triggered, indicating that there are abnormal data in the collected measurement values; otherwise, the measurement value is normal data, and the square of the residual norm of the measurement value conforms to the chi-square distribution with mn degrees of freedom. The threshold τ is determined by the hypothesis test with the confidence level mn, that is, Among them, m is the number of measured values, n is the number of state quantities; The detection rate for injection attacks is the number of detected attack vectors divided by N, where N is the number of attack vectors; Use the constructed attack scenario library to train and update the reinforcement learning agent, and update the parameters of the agent based on the current state and reward; The state space required for reinforcement learning in the reinforcement learning agent includes the system's measurement information z: voltage P, bus active power P i , reactive power P j 、Line active power P ij , reactive power Q ij , thermal limit rate rho ij ; The selection scheme of the actions of the reinforcement learning agent is as follows: the actionable space includes the beginning and end of each line, the generator and the load nodes, which are encoded by hot encoding vectors during the reinforcement learning process; The reinforcement learning agent reward function is set as: Where ρ(FDIA) represents the detection rate of false data injection attacks. If all thermal limits do not exceed 1, the reward value is the detection rate; if any line exceeds the thermal limit, the reward value is set to 0; The detection rate is the number of attack vectors exceeding the threshold among N attack vectors, z i Indicates the state when the i-th attack vector is added, a i represents the i-th attack vector, h(x i ,top t ) represents the mapping from the system topological state quantity to the quantity measurement based on time t; The training process of the reinforcement learning agent is as follows: the attacker generates an attack vector based on the initial system. At the system state at time 1, the reinforcement learning agent performs action a, the system topology changes, and the reward value is calculated. At this time, the system environment also changes. According to the new system state at time 2, the system generates a new action. This process is repeated until the reward value is stable. After the process is completed, the total reward value is recorded, and the reinforcement learning agent is updated according to the gradient descent method. Repeat the above steps until the detection rate of reinforcement learning for attack scenario library converges; Based on the trained reinforcement learning agent, a defense strategy is generated, and the operator operates the power system according to the defense strategy.

2. A mobile target defense method based on reinforcement learning according to claim 1, It is characterized in that After the defensive action occurs, the attacker still designs the attack vector based on the previous topology, which will lose concealment under the topology formed by the specific action.

3. A mobile target defense device based on reinforcement learning, It is characterized in that include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a mobile target defense method based on reinforcement learning as described in any one of claims 1-2.

4. A storage medium containing machine executable instructions, It is characterized in that The acquisition machine executable instructions are used to execute a mobile target defense method based on reinforcement learning as described in any one of claims 1-2 when executed by the acquisition machine processor.

Citation Information

Patent Citations

  • WEB dynamic adaptive defense system and defense method based on false response

    CN111917691A

  • Mobile target defense method based on flexible switching of standby line

    CN114978775A

  • Ddos attack detection and defense method for SDN environment

    WO2021227322A1