Intelligent power grid false data injection identification method based on deep reinforcement learning

Abstract

The invention discloses a smart power grid false data injection identification method based on deep reinforcement learning. The intelligent power grid false data injection identification method based on deep reinforcement learning comprises the following steps that a preset node in a power grid is selected as a reference node, a system node state set is obtained, and the system state set comprises a phase angle of a system at a detection moment and a measurement value of a detector at the detection moment; constructing a direct-current power grid system model based on the system node state set, and defining an attack strategy in the direct-current power grid system model; randomly selecting the system node state set as initial sample data, inputting the initial sample data into the direct-current power grid system model, and obtaining a countermeasure sample in combination with a deep Q learning algorithm; and building a multi-layer attack detection algorithm framework to train the adversarial sample to obtain a power grid hostile attack recognition model. According to the method, the problem of low false data attack detection precision is solved through adversarial sample generation and a multi-level attack detection algorithm framework.

Description

Technical Field

[0001] The present invention relates to the technical field of false data injection identification, and in particular to a method for identifying false data injection in a smart grid based on deep reinforcement learning. Background Art

[0002] With the integration of power grid and information communication network, cyber-physical system (CPS) has been formed, which has improved the operation capacity of power system. At the same time, this integration has also introduced new security risks. FDIA, as a means of cyber attack on power system state estimation, destroys the integrity and correctness of information by tampering with the measurement data in the SCADA system, resulting in state estimation errors and misoperation. The development of new energy Internet has made information security risks more prominent. The concealment characteristics of FDIA enable it to tamper with monitoring data and mislead control system and dispatch personnel.

[0003] Therefore, studying the FDIA detection method applicable to the new energy Internet has important practical significance for ensuring the safe and stable operation of the power information-physical system. In the process of building smart grids in my country, improving FDIA detection technology has become a key requirement to ensure the safety of the power system.

[0004] In order to cope with the new security risks brought about by the integration of power grids and information and communication networks, especially the concealment and destructiveness of false data injection attacks, it is particularly necessary to use deep reinforcement learning-based methods to simulate and detect attacks. This can not only improve the intelligence and adaptability of defense strategies, but also update attack models in real time, ensuring effective identification and defense of evolving network attacks in complex information-physical systems, and ensuring the safety and stability of the power system.

[0005] Ensemble snapshot technology is an emerging machine learning technology that originated from the research on model diversity and generalization ability in the field of deep learning. In the traditional deep learning model training process, it usually faces the problem of overfitting, that is, the model performs well on the training data, but performs poorly on unseen data. In order to solve this problem, researchers have proposed an ensemble learning method. Ensemble snapshot technology is developed on the basis of ensemble learning. It dynamically adjusts the learning rate in a single training process so that the model can converge to different local optimal solutions at different stages of training. These local optimal solutions are "snapshotted" to form a series of different models. These models are finally integrated for final prediction. This method not only reduces the computing resources required to train multiple independent models, but also improves the accuracy of prediction through model diversity.

[0006] By combining integrated snapshot technology with DNN to identify false data injection attacks, not only the accuracy and robustness of detection are enhanced through model diversity, and the system's generalization ability to unknown attacks is improved, but also real-time updates and dynamic adaptation are achieved through efficient computing processes, ensuring that the system can continue to maintain a high level of defense capabilities when facing complex and changeable attack scenarios, thereby ensuring the safe and stable operation of the power system.

[0007] When dealing with increasingly complex network attacks in power systems, especially false data injection attacks against FDIA, existing technologies face problems such as low false data attack detection accuracy, high concealment, poor adaptability, and high consumption of computing resources. Summary of the invention

[0008] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a smart grid false data injection identification method based on deep reinforcement learning, which solves the problem of low accuracy of false data attack detection through adversarial sample generation and a multi-level attack detection algorithm framework.

[0009] To achieve the above object, the present invention provides the following technical solutions: A method for identifying false data injection in a smart grid based on deep reinforcement learning comprises the following steps: selecting a preset node in a power grid as a reference node, obtaining a system node state set, wherein the system state set comprises a phase angle of the system at a detection moment and a measurement value of a detector at the detection moment; constructing a DC power grid system model based on the system node state set, and defining an attack strategy in the DC power grid system model; randomly selecting the system node state set as initial sample data and inputting it into the DC power grid system model, and obtaining adversarial samples in combination with a deep Q learning algorithm; and constructing a multi-layer attack detection algorithm framework to train the adversarial samples to obtain a power grid malicious attack recognition model.

[0010] In a preferred embodiment, the attack strategy is defined in the DC power grid system model, including defining a state space of the power grid including the phase angle state of each node at the detection moment and defining an action space of the power grid.

[0011] In a preferred embodiment, the system node state set is randomly selected as initial sample data and input into the DC power grid system model, specifically: deep Q learning is used to predict the execution of a specific action, and the model output obtains the maximum action-state function value; the action-state function value is updated according to the update rule of Q-Learning; the updated action-state function value is optimized in combination with the minimization of the loss function through an iterative process; a deep Q learning algorithm with partially different optimized parameter settings is used as an attack generator to obtain highly confusing adversarial samples.

[0012] In a preferred embodiment, the use of a partially different deep Q-learning algorithm with optimized parameter settings as an attack generator obtains highly deceptive adversarial samples, specifically: randomly generating state samples in each round, storing transfer information in a buffer, and calculating losses; obtaining classification results from a virtual environment, comparing classification labels to determine rewards; comparing misclassification probabilities, and setting rewards based on current and previous values; and dynamically adjusting the reward mechanism and combining the feedback mechanism to continuously optimize the model to generate highly deceptive adversarial samples.

[0013] In a preferred embodiment, the multi-layer attack detection algorithm framework is constructed to train the adversarial samples to obtain a power grid malicious attack identification model, specifically: setting an initial learning rate, and presetting the total number of iterations, the number of cycles, and the number of snapshots; using the cosine annealing method to dynamically adjust the initial learning rate, fluctuating from a maximum value to close to zero, and the model converges to obtain different solutions; when the verification loss of the model reaches a local minimum, the model is snapshotted to save the current parameters; after completing a snapshot, the learning rate is increased, and the next training cycle is started until the preset number of snapshots is reached; the training time of each snapshot is recorded, and the number of snapshots is adjusted according to the total training time of all snapshot models; the prediction results of all snapshot models are weighted averaged and integrated, and the SDNN algorithm is used to analyze real-time smart grid data to discover malicious activities; after detecting an attack, the FDIA data in the database is updated.

[0014] In a preferred embodiment, in the DC power grid system model, a false data attack scenario is constructed, and the false data attack scenario includes a node-based attack scenario, a time-based attack scenario, a joint node time attack scenario that considers both nodes and time, a transfer attack scenario, and a blind attack scenario.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Through a smart grid false data injection identification method based on deep reinforcement learning, a variety of attack models are collected and trained, and a dynamic attack simulation and detection framework is constructed to simulate known FDIA attack modes and generate new attack scenarios, thereby continuously improving the adaptability and foresight of the defense system, and effectively solving the problem of low accuracy of false data attack detection in existing technologies.

[0016] 2. Through the learning rate adjustment strategy in a single training process, multiple model snapshots are captured to achieve model diversity, thereby improving the generalization ability and robustness of the detection system.

[0017] 3. By integrating multiple snapshot models, the prediction accuracy is improved, while the calculation efficiency is high and the adaptability is strong, thus providing efficient and powerful security protection for the power information-physical system.

[0018] 4. By establishing a DC power grid system model that is more in line with the actual power grid environment and introducing attack strategies and multiple attack scenarios, the state changes of the power grid after being affected by different attack modes can be effectively simulated, thereby achieving more flexible and dynamic security detection of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of a method for identifying false data injection in a smart grid based on deep reinforcement learning provided in an embodiment of the present application; DETAILED DESCRIPTION The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0021] Embodiment 1, Figure 1 The present invention provides a method for identifying false data injection in a smart grid based on deep reinforcement learning, comprising the following steps: S1 selects a preset node in the power grid as a reference node, and obtains a system node state set, wherein the system state set includes a phase angle of the system at the detection time and a measurement value of a detector at the detection time; S2 constructs a DC power grid system model based on the system node state set, and defines an attack strategy in the DC power grid system model; S3 randomly selects the system node state set as initial sample data and inputs it into the DC power grid system model, and obtains adversarial samples by combining the deep Q learning algorithm; S4 builds a multi-layer attack detection algorithm framework to train the adversarial samples to obtain a power grid malicious attack recognition model.

[0022] The phase angle of the system at the detection moment is defined as follows:

[0023] The measured value of the detector at the detection moment is defined as follows:

[0024] Where: is the phase angle of the system at the detection time, It represents the phase angle of the nth node of the system at the tth detection time; is the measurement value of the detector at the detection moment, represents the measurement value of the mth detector at the tth detection time, t represents the detection time, is the total number of nodes except the reference node, is the total number of detectors in the power grid.

[0025] The DC power grid system model is specifically:

[0026] Where: To represent the phase angle of the system at the tth detection moment, represents the measurement value of the mth detector at the tth detection time, is the state transfer matrix, is the measurement matrix obtained through the structure of the power grid system, is the system noise vector at the detection time, is the measurement noise vector at the detection time, The order is The identity matrix of .

[0027] It should be noted that the system noise vector at the detection time and the measurement noise vector at the detection time obey independent Gaussian random distributions; the state transfer matrix describes the rules for the power grid to transfer from one state to another after performing a certain action. For example, after adjusting the power of a certain transmission line, the load distribution, current, and voltage in the power grid may change; the measurement matrix is ​​used to obtain data from various sensors in the power grid, or to evaluate the status of each node and line in the power grid.

[0028] Constructing false data attack scenarios based on the DC power grid system model, wherein the false data attack scenarios include node-based attack scenarios, time-based attack scenarios, joint node-time attack scenarios that consider both nodes and time, transfer attack scenarios, and blind attack scenarios; In the node-based attack scenario, the attacker may select one or more components and use them as attack targets to obtain a first system state after the attack; The first system state after the attack is as follows:

[0029] Where: is the first system state after the attack, is the attack function, is the weight coefficient of each item that determines the original state value in the false state value. is the exponential coefficient that determines how each term of the original state value is transformed, is the constant false state value injected by the attacker directly added to the transformed state. represents the number of times the attacker performs different transformations and combinations on the original state value, t represents the detection time, is the total number of nodes except the reference node.

[0030] In the time-based attack scenario, Attack at the moment state and obtain the second attacked system state; the second attacked system state is as follows:

[0031] Where: is the first system state after the attack, is the weight coefficient of each item that determines the original state value in the false state value. is the exponential coefficient that determines how each term of the original state value is transformed, is the constant false state value injected by the attacker directly added to the transformed state, t represents the detection time, is the total number of nodes except the reference node.

[0032] The joint node time attack scenario that considers both nodes and time; In the described transfer attack scenario, the attacker does not inject false data, but shifts the actual state data in time by changing the measurement time point; In the blind attack scenario, the attacker randomly injects false state values, most of whose elements are zero or close to zero.

[0033] Determine the environment of the power grid model, including power generation units, loads, transmission lines and other components in the power grid. Sample the phase angles and measurements of each node in the power grid every 15 minutes, and randomly select samples from the historical data of the power grid to ensure that a certain proportion of abnormal or attack situations are included in the sample set.

[0034] Defining the attack strategy in the DC power grid system model includes defining a state space of the power grid including the phase angle state of each node at the detection moment and defining an action space of the power grid; The definition of the power grid includes the state space of the phase angle state of each node at the detection time, which is as follows:

[0035] The action space of the defined power grid includes actions such as adjusting power generation, changing load distribution, switching line status, etc., as follows:

[0036] Where: is the state space of the phase angle state of the node at the detection moment, is the action space of the power grid, The representative node increases the power generation, The representative node reduces the power generation, The representative node disconnects the line. represents the node maintaining the current state, t represents the detection time, is the total number of nodes except the reference node.

[0037] The randomly selecting the system node state set as initial sample data and inputting it into the DC power grid system model is specifically: Use deep Q learning to predict specific actions, and the model outputs the maximum action-state function value; Updating the action-state function value according to the Q-Learning update rule; The updated action-state function value is combined with the minimization loss function to optimize the parameters through an iterative process; A deep Q-learning algorithm with partially different optimized parameter settings is used as the attack generator to obtain highly deceptive adversarial samples.

[0038] The working principle of Q-Learning is as follows: Action-state function value (Q-value): In reinforcement learning, the action-state function value represents the maximum long-term reward that can be obtained by performing an action in a specific state. The action-state function value is the basis of the agent's decision-making, which measures the "goodness" of taking an action in a certain state.

[0039] State: Indicates the current state of the environment. In the power grid scenario, the state may include the current, voltage, load conditions, etc. of each node in the power grid.

[0040] Action: An action that an agent can take. For example, in a power grid, an action might be to adjust the power output of a transmission line or change the voltage at a node.

[0041] Reward: The feedback value returned by the environment to the agent after performing an action. This reward can be positive (for example, improving system stability or reducing energy consumption). ) or negative (e.g. increasing system instability or violating safety limits).

[0042] The update rule of Q-Learning is as follows:

[0043] In the formula: s represents the state, a represents the action, Indicates the value of the action in the state at the detection time, Indicates the transition from one state to the next via an action. represents the learning rate, represents the discount factor, r represents the immediate reward received from the environment after executing the action, and the maximum action-state function value of all possible actions in the new state.

[0044] The minimization loss function is as follows:

[0045] Where: To minimize the loss function, are the parameters of the agent, Represents the experience buffer The expected value of the sample.

[0046] The deep Q learning algorithm with partially different optimized parameter settings is used as an attack generator to obtain highly confusing adversarial samples. The input includes a reply buffer, a capacity limit, a mini-batch size, and a learning rate, specifically: In each round, randomly generate state samples, store transition information in a buffer, and calculate the loss; Get the classification results from the virtual environment and compare the classification labels to determine the reward: if the labels match, the reward is set to the median value; otherwise, the reward is 0; Compare the misclassification probabilities and set the reward based on the current and previous values: if the new value is higher, the reward is low; otherwise, the reward is high. In this process, the reward mechanism is dynamically adjusted to enable the attacker to continuously learn and adapt to changes in the virtual environment. At the same time, the feedback mechanism is used to continuously optimize the model to generate highly confusing adversarial samples.

[0047] In Deep Q Learning (DQN), the action-state function value is not represented by a simple table, but a deep neural network is used to approximate the action-state value function. The advantage of this is that when the state space is very large (such as the state space in the power grid), we can effectively approximate the action-state function value through a neural network without enumerating all possible state and action pairs.

[0048] The multi-layer attack detection algorithm framework is constructed to train the adversarial samples to obtain a power grid malicious attack recognition model, specifically: Set the initial learning rate, and preset the total number of iterations, cycles, and snapshots; The initial learning rate is dynamically adjusted using a cosine annealing method, fluctuating from a maximum value to close to zero, and the model converges to obtain different solutions; When the validation loss of the model reaches a local minimum, the model is snapshotted to save the current parameters; After completing a snapshot, the learning rate is increased and the next training cycle begins until the preset number of snapshots is reached; Record the training time of each snapshot, and adjust the number of snapshots according to the total training time of all snapshot models; The prediction results of all snapshot models are weighted averaged and integrated, and the SDNN algorithm is used to analyze real-time smart grid data to detect malicious activities; Update the FDIA data in the database after an attack is detected.

[0049] The cosine annealing method is used to dynamically adjust the initial learning rate, specifically:

[0050] Where: The adjusted learning rate, is the initial learning rate, is the preset total number of iterations, The preset number of cycles.

[0051] The method of adjusting the number of snapshots according to the total training time of all snapshot models is specifically as follows:

[0052] Where: is the number of snapshots after adjustment, is the training time for each snapshot recorded, is the preset number of snapshots, is the standard training time for a deep neural network.

[0053] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0054] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0055] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0056] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0057] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0058] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying false data injection in smart grid based on deep reinforcement learning, characterized in that: The following steps are involved: Selecting a preset node in the power grid as a reference node, and obtaining a system node state set, wherein the system state set includes a phase angle of the system at the detection time and a measurement value of a detector at the detection time; Based on the system node state set, a DC power grid system model is constructed, and an attack strategy is defined in the DC power grid system model; Randomly selecting the system node state set as initial sample data and inputting it into the DC power grid system model, and combining it with a deep Q learning algorithm to obtain adversarial samples; A multi-layer attack detection algorithm framework is built to train the adversarial samples to obtain a power grid malicious attack recognition model.

2. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 1, characterized in that: Defining the attack strategy in the DC power grid system model includes defining a state space of the power grid including the phase angle state of each node at the detection moment and defining an action space of the power grid; The definition of the power grid includes the state space of the phase angle state of each node at the detection time, which is as follows: The action space of the power grid is defined as follows: Where: is the state space of the phase angle state of the node at the detection moment, is the action space of the power grid, The representative node increases the power generation, The representative node reduces the power generation, The representative node disconnects the line. represents the node maintaining the current state, t represents the detection time, is the total number of nodes except the reference node.

3. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 2, characterized in that: The randomly selecting the system node state set as initial sample data and inputting it into the DC power grid system model is specifically: Use deep Q learning to predict specific actions, and the model outputs the maximum action-state function value; Updating the action-state function value according to the Q-Learning update rule; The updated action-state function value is combined with the minimization loss function to optimize the parameters through an iterative process; A deep Q-learning algorithm with partially different optimized parameter settings is used as the attack generator to obtain highly deceptive adversarial samples.

4. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 3, characterized in that: The deep Q learning algorithm with partially different optimized parameter settings is used as an attack generator to obtain highly confusing adversarial samples, specifically: In each round, randomly generate state samples, store transition information in a buffer, and calculate the loss; Get the classification results from the virtual environment and compare the classification labels to determine the reward; Compare the misclassification probabilities and set the reward based on the current and previous values; By dynamically adjusting the reward mechanism and combining it with the feedback mechanism, the model is continuously optimized to generate highly confusing adversarial samples.

5. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 4, characterized in that: The multi-layer attack detection algorithm framework is constructed to train the adversarial samples to obtain a power grid malicious attack recognition model, specifically: Set the initial learning rate, and preset the total number of iterations, cycles, and snapshots; The initial learning rate is dynamically adjusted using a cosine annealing method, fluctuating from a maximum value to close to zero, and the model converges to obtain different solutions; When the validation loss of the model reaches a local minimum, the model is snapshotted to save the current parameters; After completing a snapshot, the learning rate is increased and the next training cycle begins until the preset number of snapshots is reached; Record the training time of each snapshot, and adjust the number of snapshots according to the total training time of all snapshot models; The prediction results of all snapshot models are weighted averaged and integrated, and the SDNN algorithm is used to analyze real-time smart grid data to detect malicious activities; Update the FDIA data in the database after an attack is detected.

6. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 5, characterized in that: The DC power grid system model is specifically: Where: To represent the phase angle of the system at the tth detection moment, represents the measurement value of the mth detector at the tth detection time, is the state transfer matrix, is the measurement matrix obtained through the structure of the power grid system, is the system noise vector at the detection time, is the measurement noise vector at the detection time, The order is The identity matrix of .

7. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 6, characterized in that: In the DC power grid system model, false data attack scenarios are constructed, including node-based attack scenarios, time-based attack scenarios, joint node-time attack scenarios that consider both nodes and time, transfer attack scenarios, and blind attack scenarios.

8. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 7, characterized in that: The update rule of Q-Learning is as follows: In the formula: s represents the state, a represents the action, Indicates the value of the action in the state at the detection time, Indicates the transition from one state to the next via an action. represents the learning rate, represents the discount factor, r represents the immediate reward received from the environment after executing the action, and the maximum action-state value function value of all possible actions in the new state.

9. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 8, characterized in that: The cosine annealing method is used to dynamically adjust the initial learning rate, specifically: Where: The adjusted learning rate, is the initial learning rate, is the preset total number of iterations, The preset number of cycles.

10. A method for identifying false data injection in a smart grid based on deep reinforcement learning as claimed in claim 9, characterized in that: The method of adjusting the number of snapshots according to the total training time of all snapshot models is specifically as follows: Where: is the number of snapshots after adjustment, is the training time for each snapshot recorded, is the preset number of snapshots, is the standard training time for a deep neural network.

Images