A method for detecting false data injection attacks in power networks

Abstract

The present invention discloses a method for detecting false data injection attacks in power networks, comprising: establishing a deep learning model for the environment in a reinforcement learning manner, interacting with an intelligent agent and the environment, constructing a reinforcement learning multi-tuple based on a data set collected by the power data acquisition and monitoring control system, and collecting the multi-tuple into an experience pool as a sampling trajectory for subsequent training; inputting the sampling trajectory in the experience pool into an actor intelligent agent and a critic intelligent agent respectively, training the corresponding intelligent agents through a long short-term memory network module and a multi-layer linear network module, and then using a loss function to enable the actor intelligent agent and the critic intelligent agent to interact and cooperate to update their respective network parameters to optimize the data detection function of the deep learning model. The detection method provided by the present invention utilizes the long short-term memory network module to also use historical state input as the basis for detection, thereby improving the system's detection and recognition accuracy for injection attacks.

Description

Technical Field

[0001] The present invention relates to the technical field of network data security, and in particular to a method for detecting false data injection attacks on power networks. Background Art

[0002] With the rapid development of society and the use of renewable energy, electricity consumption has increased dramatically. As the foundation of the national economy, the State Grid bears the responsibility of maintaining the national energy system and social development. To meet increasingly diverse user needs and enhance intelligence, many countries around the world have proposed plans and programs to transform traditional power grids into smart grids. A smart grid is a fully automated power transmission network that monitors and controls every user and grid node, ensuring the bidirectional flow of information and power between all nodes throughout the entire transmission and distribution process, from power plants to end users. By integrating power, information, and business flows, a smart grid can rationally distribute power quality to meet user needs and reduce the overall operating costs of the system.

[0003] However, most existing smart grids adopt an architecture where information and power networks are interdependent. In this case, local malicious attacks can threaten the security of the entire power system. False data injection attacks are a relatively common malicious attack. Because their purpose is to destroy data integrity, they can bypass traditional state monitoring mechanisms and directly interfere with the decision-making of the power grid control center. False data injection attacks pose a significant threat to the security of current smart grids. Currently, the detection of false data attacks mainly adopts the method of state estimation and difference judgment, which determines whether the data is erroneous by calculating the difference before and after the state prediction. However, this method only uses single-step features for judgment, ignoring the time series characteristics of the data, and therefore cannot effectively detect attacks. Especially in power networks, due to the time series characteristics of their data, there is an urgent need for a false data injection attack detection method that can fully utilize time series data.

[0004] The maximum entropy offline actor-critic method combines the offline strategy, the actor-critic method, and maximum entropy. It uses the offline strategy to improve sample efficiency and uses maximum entropy to ensure stability and exploration. Therefore, the method is effective and highly stable.

[0005] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention provides a method for detecting false data injection attacks in power networks. The specific technical solution is as follows:

[0007] In one aspect, a method for detecting false data injection attacks on power networks is provided, comprising the following steps: establishing a deep learning model for an environment using reinforcement learning, wherein the reinforcement learning employs an offline actor-critic method based on maximum entropy to iteratively update the model, and the environment is a power system attacked by false data injection; in the deep learning model, an intelligent agent interacts with the environment, constructs a reinforcement learning multi-tuple based on a data set collected by a power data acquisition and monitoring control system, and collects the multi-tuple into an experience pool as a sampling trajectory for subsequent training;

[0008] The multi-tuple includes a state value, an agent's judgment, a reward value, and a next state, wherein the state value includes multiple input features in the data set, the agent's judgment includes whether the current power system is under attack, and after the agent makes a decision, the power system enters the next state. The reward value is the corresponding reward obtained by the agent in the interaction;

[0009] The intelligent agent includes an actor agent and a critic agent, and both the actor agent and the critic agent include a long short-term memory network module and a multi-layer linear network module; the sampling trajectories in the experience pool are respectively input into the actor agent and the critic agent, and the corresponding intelligent agents are trained by the long short-term memory network module and the multi-layer linear network module, and then the actor agent and the critic agent are interactively cooperated through the loss function to update their respective network parameters, so as to optimize the data detection function of the deep learning model.

[0010] Furthermore, the environment is a false data injection attack against automatic voltage control. The state value is the observed value of the power system, which includes the voltage amplitude on the bus, the bus power injection value, the real power between nodes, and the reactive power. The corresponding reward obtained by the agent's decision is calculated as follows:

[0011] A. If the data is tampered with, the agent is considered to be under attack and receives a reward value a.

[0012] B. If the data is tampered with and the agent determines that it has not been attacked, the agent receives a reward value b;

[0013] C. If the data has not been tampered with, the agent is considered to have been attacked and receives a reward value c.

[0014] D. If the data has not been tampered with, the agent determines that it has not been attacked, and the agent receives a reward value d;

[0015] Among them, a>d>0, c <b<0。

[0016] Furthermore, in the long short-term memory network module, the corresponding input is the state value z t And the retained features h of historical input information t-1 , the corresponding output is the extracted feature p t , and according to the extracted features p t Get the retained feature h of the historical information corresponding to this output t .

[0017] Furthermore, a multi-layer linear network module is used to continue extracting features and making judgments on the features extracted by the long short-term memory network module. According to the discrete attributes of the judgment, the output of the final actor is a classification distribution. Sampling is performed from the classification distribution to obtain the judgment of the intelligent agent on whether the current data has been tampered with. After executing the judgment, the environment will return to the next state.

[0018] Furthermore, if the set of trajectories sampled from the experience pool each time is B, then the loss function of the actor agent is composed of the critic's Q-value evaluation of the state-action pair and the entropy of the actor's output, and its calculation formula is as follows:

[0019]

[0020] where ψ i Corresponding to the i-th critic network, α corresponds to the entropy of the network output distribution, The output of the actor network when the corresponding input state is z.

[0021] Furthermore, the loss function calculation formula of the critic agent is as follows:

[0022]

[0023] Among them, γ represents the discount coefficient, V corresponds to state z t+1 value.

[0024] Furthermore, after calculating the loss according to the loss functions of the actor agent and the critic agent, the gradient is calculated using the gradient descent method to guide the update of the network parameters in the agent.

[0025] Furthermore, the actor agent and the critic agent have the same internal structure, but their input and output dimensions are different.

[0026] On the other hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned detection method when executing the computer program.

[0027] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned detection method is implemented.

[0028] Compared with the existing technology, the present invention has the following advantages: it uses reinforcement learning to detect attacks, adds a long-short-term memory network module to the network, and uses historical state input as the basis for detection to improve the system's detection and recognition accuracy for injection attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the model architecture of the power network false data injection attack detection method provided by an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of the internal structure of an actor agent in the method for detecting false data injection attacks on power networks provided by an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the internal structure of the critic agent in the power network false data injection attack detection method provided by an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the framework of the long short-term memory network module in the method for detecting false data injection attacks on power networks provided by an embodiment of the present invention;

[0033] Figure 5 The present invention provides a flow chart of a method for detecting false data injection attacks in a power network. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and the accompanying drawings are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0036] Reinforcement learning is a trial-and-error machine learning method that gradually improves the performance of an agent through interaction with the environment. Compared to traditional machine learning methods, reinforcement learning does not require accurate labels; instead, it only requires a specifically designed reward function to guide the agent's training direction. The maximum entropy-based offline actor-critic method is a reinforcement learning method that optimizes the policy to achieve higher cumulative rewards while maximizing the policy's entropy to prevent the agent from falling into a local optimum. Furthermore, the soft actor-critic method employs a dual-critic network architecture, which effectively prevents the agent from falling into the overvaluation problem. Furthermore, this method does not require prior knowledge of the environment's state transitions and can solve a range of random environment problems.

[0037] In one embodiment of the present invention, a method for detecting false data injection attacks in a power network is provided, comprising the following steps: establishing a deep learning model for an environment in a reinforcement learning manner, wherein the reinforcement learning adopts an offline actor-critic method based on maximum entropy to iteratively update the model, wherein the environment is a power system attacked by false data injection; in the deep learning model, see Figure 1 , the intelligent agent interacts with the environment, constructs a multi-tuple for reinforcement learning based on the data set collected by the power data acquisition and monitoring control system, and collects the multi-tuple into the experience pool as a sampling trajectory for subsequent training;

[0038] The multi-tuple includes a state value, an agent's judgment, a reward value, and a next state, wherein the state value includes multiple input features in the data set, the agent's judgment includes whether the current power system is under attack, and after the agent makes a decision, the power system enters the next state. The reward value is the corresponding reward obtained by the agent in the interaction;

[0039] The agent includes an actor agent and a critic agent. The actor agent and the critic agent have the same internal structure, but their input and output dimensions are different. Figure 2 and Figure 3 The actor agent and the critic agent both include a long short-term memory network module (LSTM) and a multilayer linear network module (MLP). The sampling trajectories in the experience pool are respectively input into the actor agent and the critic agent, and the corresponding agents are trained by the long short-term memory network module and the multilayer linear network module. Then, the actor agent and the critic agent are made to interact and cooperate through the loss function to update their respective network parameters, so as to optimize the data detection function of the deep learning model.

[0040] This embodiment models the environment using reinforcement learning, replacing the previous single-step state estimation method. It adopts the maximum entropy-based offline actor-critic method in reinforcement learning to detect false data injection attacks, and uses the sequence relationship between data to determine whether the system is under attack, thereby improving the system's detection and recognition accuracy for injection attacks.

[0041] Specifically, this embodiment includes the following steps:

[0042] (1) Modeling and initialization

[0043] First, the environment is modeled using reinforcement learning, with the multi-tuple set as a quaternary. Taking the automatic voltage control based on the optimal power flow model as an example, the definition of the corresponding quaternary (z, y, b, z') in reinforcement learning is given, where the state value (z) is the observation value of the power system, specifically including the voltage amplitude on the bus, the bus power injection value, the real power and reactive power between nodes, and the decision information made by the optimal power flow model; the agent judgment (y) is the agent's judgment on whether the current system is under attack; the next state (z') is the combination of the next state the power system enters after the optimal power flow model makes a decision and the next decision of the optimal power flow model; the reward value (b) is the reward obtained by the agent in this interaction, which is calculated as follows:

[0044] If the data is tampered with, the agent is considered to be under attack and receives a +1 reward.

[0045] If the data is tampered with, the agent is judged as not attacked and receives a -1 reward;

[0046] The data has not been tampered with, but the agent is considered to have been attacked and receives a -10 reward.

[0047] The data has not been tampered with, the agent determines that it has not been attacked, and the agent receives a reward of 0.1.

[0048] Then, the actor network φ(z) in the actor agent, the critic network ψ(z, y) in the critic agent, and the experience pool U are initialized. Considering the implicit time series relationship of the observations in the environment, the design of both the actor and critic agents includes long short-term memory network modules that consider historical input features.

[0049] (2) Data Collection

[0050] Next, we use the actor network to interact with the simulation environment to collect data. After the state value of the environment is passed into the network, it first passes through the LSTM module, see Figure 4 , whose corresponding input is the state value z t And the retained features h of historical input information t-1, and the output is the extracted feature p t There are three units in the LSTM module: input gate, forget gate and output gate. Formulas (1)-(6) describe the transmission process of the network in detail. Formula (1) describes the transmission of the forget gate. This layer determines which features should be discarded from the cell state. In the formula, W l with b l Represent the weight and bias of the network layer respectively, and σ corresponds to the sigmoid activation function; Formula (2) and Formula (3) describe the transmission of the input gate. This layer determines what information should be saved in the cell state. Formula (2) describes what information needs to be updated, while Formula (3) completes the update of the corresponding cell state. In the formula, W e with b e Represent the weight and bias of the updated information network, W N with b N Represent the weight and bias of the updated information network respectively, and tanh corresponds to the hyperbolic tangent activation function; Formula (4) and Formula (5) correspond to the transmission of the output gate. After the cell state is processed by the hyperbolic tangent activation function tanh and then multiplied by the output value obtained by Formula (3), the final extracted feature p can be obtained. t , W in the formula p with b p Represent the weight and bias of the network layer, p t Corresponding to the features extracted by the final network, h t The corresponding historical information of this process is retained.

[0051] f t =σ(W l ×[h t-1 ,z t ]+b l ) (1)

[0052] e t =σ(W e ×[h t-1 ,z t ]+b e ) (2)

[0053] N t =f t *N t-1 +e t *(tanh(W N ×[h t-1 ,z t ]+b N )) (3)

[0054] p t =σ(W p ×[ht-1 ,z t ]+b p ) (4)

[0055] h t =p t *tanh(N t ) (5)

[0056] Next, a multi-layer linear network is used to continue extracting features and making judgments. Since the judgment is a discrete attribute, the final output of the actor is a categorical distribution. Sampling from this distribution can then obtain the agent's judgment y on whether the current data has been tampered with. After executing the judgment, the environment will return to the next state z', and then the four-tuple (z, y, b, z') can be stored in the experience pool U.

[0057] During the collection process, the decision-making of the intelligent agent is completed by extracting time series features through LSTM and then using a reinforcement learning model.

[0058] (3) Update the network

[0059] When the set number of data is collected, it can be sampled from the experience pool and the network can be updated. Considering the temporal relationship of information, sampling is performed in segments. A segment includes multiple temporal point data. Assuming that the trajectory set of each sampling is B, the corresponding actor loss function is calculated as shown in formula (6). The loss is composed of the critic's Q-value evaluation of the state-action pair and the entropy of the actor's output, where ψ i Corresponding to the i-th critic network, α corresponds to the entropy of the network output distribution to avoid overfitting of the agent, y φ (z) corresponds to the output of the actor network when the input state is z. The calculation method is shown in formula (7). First, the distribution function p of the corresponding probability is constructed according to the output of the network, and then the corresponding action is sampled from p. The function f corresponds to the process of constructing the distribution and sampling.

[0060]

[0061]

[0062] Correspondingly, we can also obtain the critic’s loss function as shown in the following formula (8), where γ represents the discount coefficient, which measures the impact of the value of the next state on the current state; V corresponds to the state z t+1 The value of can be further expanded into the actor's t+1 The estimated Q value obtained when .

[0063]

[0064] After calculating the loss according to the above loss function, the gradient is calculated using the gradient descent method, and then the model parameters are updated using the smoothing method shown in formula (9), where and preliminary Update to get new Use the new Participate in the next update.

[0065]

[0066] Among them, when the optimization update just starts, the gradient is calculated through the loss function, and the optimizer is used to update the network parameters accordingly based on the information of gradient descent. After multiple adjustments, the gradient will gradually approach 0 until the network parameters no longer change, indicating that the optimal network parameters are reached at this time and the training is completed.

[0067] In the update network, we use the maximum entropy-based offline actor-critic method from reinforcement learning to iteratively update the model. It should be noted that collecting data into the experience pool and updating the network can be done simultaneously, or they can be done alternately in steps.

[0068] In one embodiment of the present invention, see Figure 5 After environment modeling and network initialization, the quadruple data is collected into the experience pool. When the experience pool meets the training conditions, the neural network training is performed by sampling from the experience pool to update the model until the model converges. Otherwise, it is necessary to continue collecting data to enrich the content of the training set. When the model converges, the model is output.

[0069] On the basis of the above embodiments, an optimized model is obtained, and then a more accurate judgment is obtained, so that decisions can be made accurately. If the judgment result belongs to the preset alarm condition, it is considered that an attack is discovered, an alarm is issued, and network security is improved.

[0070] An embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned detection method when executing the computer program. The concept underlying this embodiment of the device is identical to the working process of the detection method in the aforementioned embodiment, and the entire contents of the aforementioned detection method embodiment are incorporated herein by reference and will not be further elaborated.

[0071] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned detection method. The concept underlying this storage medium embodiment is consistent with the working process of the detection method described in the aforementioned embodiment. The entire contents of the aforementioned detection method embodiment are hereby incorporated by reference into this storage medium embodiment and will not be further elaborated upon.

[0072] The detection method provided by the present invention is a false data injection attack detection method for power networks based on maximum entropy offline actor-criticism. Different from the traditional difference-based detection method, it adopts reinforcement learning method for attack detection. By modeling the environment and extracting time series input features, and adding a long short-term memory network module to the network, historical state input is also used as the basis for detection.

[0073] The above description is only a preferred embodiment of the present invention and does not limit the scope of the patent. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, directly or indirectly applied in other related technical fields, is also included in the scope of patent protection of the present invention.

Claims

1. A method for detecting false data injection attacks in power networks, characterized in that: The following steps are involved: A deep learning model is established for an environment using reinforcement learning, wherein the reinforcement learning employs an offline actor-critic method based on maximum entropy to iteratively update the model. The environment is a power system attacked by false data injection. In the deep learning model, an intelligent agent interacts with the environment, constructs a reinforcement learning multi-tuple based on a data set collected by a power data acquisition and monitoring control system, and collects the multi-tuple into an experience pool as a sampling trajectory for subsequent training. The multi-tuple includes a state value, an agent's judgment, a reward value, and a next state, wherein the state value includes multiple input features in the data set, the agent's judgment includes whether the current power system is under attack, and after the agent makes a decision, the power system enters the next state. The reward value is the corresponding reward obtained by the agent in the interaction; The intelligent agent includes an actor agent and a critic agent, and the actor agent and the critic agent both include a long short-term memory network module and a multi-layer linear network module; the sampling trajectories in the experience pool are respectively input into the actor agent and the critic agent, and the corresponding intelligent agents are trained by the long short-term memory network module and the multi-layer linear network module, and then the actor agent and the critic agent are interactively cooperated by using a loss function to update their respective network parameters, so as to optimize the data detection function of the deep learning model; Among them, if the trajectory set sampled from the experience pool each time is B, then the loss function of the actor agent is composed of the critic's Q-value evaluation of the state-action pair and the entropy of the actor's output. Its calculation formula is as follows: Among them, ψ i Corresponding to the i-th critic network, α corresponds to the entropy of the network output distribution, The output of the actor network when the corresponding input state is z; The loss function calculation formula of the critic agent is as follows: Among them, γ represents the discount coefficient, V corresponds to state z t+1 value.

2. The method for detecting false data injection attacks in a power network according to claim 1, characterized in that: The environment is a false data injection attack on automatic voltage control. The state value is the observation value of the power system, which includes the voltage amplitude on the bus, the bus power injection value, the real power between nodes, and the reactive power. The corresponding reward obtained by the decision of the actor agent is calculated as follows: A. If the data is tampered with, the actor agent is determined to be under attack, and the actor agent receives a reward value a; B. If the data is tampered with and the actor agent determines that it has not been attacked, the actor agent receives a reward value b; C. If the data has not been tampered with, the actor agent determines that it has been attacked, and the actor agent receives a reward value c; D. If the data has not been tampered with, the actor agent determines that it has not been attacked, and the actor agent receives a reward value d; Among them, a>d>0, c <b<0。 3. The method for detecting false data injection attacks in a power network according to claim 1, wherein: In the long short-term memory network module, the corresponding input is the state value z t And the retained features h of historical input information t-1 , the corresponding output is the extracted feature p t , and according to the extracted features p t Get the retained feature h of the historical information corresponding to this output t .

4. The method for detecting false data injection attacks in a power network according to claim 3, characterized in that: The multi-layer linear network module is used to continue extracting features and making judgments on the features extracted by the long short-term memory network module. According to the discrete attributes of the judgment, the final output of the actor is a classification distribution. Sampling is performed from the classification distribution to obtain the judgment of the actor agent on whether the current data has been tampered with. After executing the judgment, the environment will return to the next state.

5. The method for detecting false data injection attacks in a power network according to claim 1, wherein: After calculating the loss according to the loss function of the actor agent and the critic agent, the gradient is calculated using the gradient descent method to guide the update of the network parameters in the actor agent and the critic agent.

6. The method for detecting false data injection attacks in a power network according to claim 1, characterized in that: The actor agent and the critic agent have the same internal structure, but the dimensions of their input and output are different.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the detection method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the detection method according to any one of claims 1 to 6 is implemented.

Images