A time series prediction reinforcement learning detection method for covert attacks on smart grids

Abstract

The present invention belongs to the technical field of electronic digital data processing, and more specifically, relates to a time series prediction reinforcement learning detection method for covert attacks on smart grids. The method comprises the following steps: first, modeling the smart grid, obtaining instrument measurement data, and preprocessing the data; second, using the preprocessed data as the input of a long short-term memory network to estimate the state of the smart grid; then, modeling the attack detection problem in the smart grid as a partially observable Markov decision problem; finally, using a reinforcement learning method to solve the partially observable Markov decision problem to achieve the detection of covert attacks on the smart grid. The present invention can detect covert attacks in smart grids with lower latency and false detection rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic digital data processing, and more specifically, relates to a time series prediction reinforcement learning detection method for covert attacks on smart grids. Background Art

[0002] The massive sensors and new measurement and control methods integrated in modern power systems allow for efficient data transmission and information interaction within the system. Smart grids achieve information perception, collection, processing and application throughout the entire life cycle from the supply side to the load side by applying modern information technology, communication technology and control technology. These functions include status monitoring, data analysis, prediction and early warning of smart grid equipment. The massive data generated provides the basis for the intelligent management of smart grids. Large-scale sensor data can improve energy efficiency, reliability and manageability, and guide the control center to make correct decisions. However, this has resulted in a large number of devices and access points being located outside the traditional management domain, breaking the security boundaries of traditional basic measures. Vulnerabilities in the operation and monitoring modules of smart grids can be exploited by malicious attackers to cause the system's operating state to shift, thereby affecting the normal operation of the system and subsequent data mining and utilization.

[0003] Hidden attacks can bypass the system's anomaly detection mechanism and cause damage to the system. They have multiple modes and strong system adaptability, and are a risk to the safe operation of smart grid systems that can pose a serious threat. Traditional bad data detection technology relies on linear estimators and residual thresholds to detect malicious attacks, and uses the last moment's observation data to estimate the operating state, so it is difficult to deal with hidden attacks. In order to better utilize the massive and complex historical data in smart grids, in the complex and changeable smart grid environment, reinforcement learning interacts with the environment in real time through terminals, and continuously explores decisions that make the environment safer and safer through rewards or punishment measures. This is more suitable for solving practical problems of smart grids. Existing reinforcement learning-based methods have achieved remarkable results in the problem of attack detection in smart grids, but there are still the following problems: the existing reinforcement learning-based detection algorithm discretizes the system state and only uses the current state information to determine whether there is an attack, completely ignoring the continuous change characteristics of the smart grid state. To this end, the sliding window technology is introduced to use multiple historical state information to determine whether there is an attack.

[0004] Furthermore, in order to more accurately construct a model of nonlinear power systems, the long short-term memory network (LSTM) is used to characterize the dynamics of the power grid. The long short-term memory network can finely capture the complex nonlinear relationship between the measured data and the real-time state, thereby significantly improving the estimation accuracy of the system state, while retaining the temporal dependency information in the time series data, providing strong support for the dynamic monitoring and regulation of smart grids. In order to capture the continuous dynamic changes contained in the power grid state, the sliding window technology is introduced. It effectively reduces the uncertainty in the smart grid state judgment problem by integrating and analyzing the state information in the observation history. Subsequently, reinforcement learning is used to map this window containing rich historical information to a specific operation or decision-making problem, thereby achieving effective detection of hidden attacks in smart grids.

[0005] Chinese patent document CN113268730A discloses a method for detecting false data injection attacks in smart grids based on reinforcement learning. The method is based on the detection method of false data injection attacks of the Sarsa algorithm, and the attacks are divided into direct false data injection attacks and hidden false data injection attacks for separate detection. In terms of observation value construction, the direct false data injection attack detection combines the residual method with the threshold segmentation, and the hidden false data injection attack combines the difference norm of the measured value with the threshold segmentation to obtain the observed value. The Q table is obtained by training the observed values, and the attack is detected using the Q table. However, the measured data and the implementation state in the power grid are usually nonlinear, and the method cannot accurately capture the complex nonlinear association between the measured data and the real-time state of the power grid using a linear estimator. In addition, when the method uses the observed value to train the Q table, it only considers the detection rate to update the Q table, and does not consider the impact of the detection delay on the attack detection in the real-time operation of the power grid when the power grid is running in real time.

[0006] In view of this, the present invention designs a time series prediction reinforcement learning detection method for covert attacks on smart grids. Summary of the invention

[0007] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a time series prediction reinforcement learning detection method for covert attacks on smart grids, detect various covert attacks on smart grids, and reduce the impact of attacks on smart grids with lower detection delay; SUMMARY OF THE INVENTION

[0009] The present invention provides a time series prediction reinforcement learning detection method for covert attacks on smart grids. The long short-term memory network (LSTM) is used to estimate the state of the smart grid, and reinforcement learning is used to detect whether there is a covert attack. First, the real-time state of the smart grid is estimated by the long short-term memory network, and then the residual between the estimated value and the measured value is calculated, and the historical residual is recorded. A Q value table is trained using reinforcement learning, and a judgment is made on whether there is an attack on the current smart grid based on the current residual and the recorded historical residual.

[0010] The detailed technical scheme of the present invention is as follows:

[0011] A time series prediction reinforcement learning detection method for covert attacks on smart grids, the method comprising:

[0012] S1. Model the smart grid, obtain instrument measurement data, and pre-process the data;

[0013] S2, using the preprocessed data as the input of the long short-term memory network to estimate the state of the smart grid and determine whether there is an attack;

[0014] S3, the problem of using the estimated power grid state to determine whether there is an attack is modeled as a partially observable Markov decision problem;

[0015] S4. Solve the partially observable Markov decision problem based on reinforcement learning and realize the detection of hidden attacks on smart grids.

[0016] According to the preferred embodiment of the present invention, the modeling of the smart grid in S1 and obtaining the instrument measurement data refers to: the smart grid includes Busbars and Instruments, , taking one of the busbars as the reference busbar, Smart grid The state vector at the moment is as follows:

[0017] (1)

[0018] In formula (1), represents the state value at bus n at time t; T is the transpose;

[0019] Smart grid The measurement vector at the moment is as follows:

[0020] (2)

[0021] In formula (2), Indicates at time The measured value at the instrument k; T is the transpose;

[0022] Based on the widely used nonlinear DC power system model, the smart grid is modeled with the following state-space equations:

[0023] (3)

[0024] (4)

[0025] In formula (3) and (4), is the system matrix, is the measurement matrix determined based on the network topology, is the problem noise vector, is the measurement noise vector.

[0026] According to a preferred embodiment of the present invention, the preprocessed data includes instrument measurement values and actual value of the power grid :

[0027] The preprocessing process is: obtain T instrument measurement values ​​in continuous time and calculate their mean vector and the standard deviation vector , then at time The measurement vector at After processing, it is expressed as

[0028] (5)

[0029] Get the actual values ​​of T power grids in continuous time and calculate their mean vector and the standard deviation vector , then at time The state vector at After processing, it is expressed as

[0030] (6).

[0031] Preferably, according to the present invention, the long short-term memory network comprises the following six-layer structure:

[0032] The first layer is the sequence input layer: the instrument measurement values ​​in the preprocessed data are converted into sequence format as input data and passed to the next layer for processing. The feature dimension of the input sequence is equal to the dimension of the measurement value.

[0033] The second layer is the long short-term memory network layer: the core layer of the model processes time series, models them by learning the temporal patterns and dependencies in the input sequence, and adjusts the parameters of the long short-term memory network model through the back-propagation algorithm to minimize the error between the prediction and the actual value; through continuous iterative training and optimization, the long short-term memory network model gradually improves its prediction performance, so that it can better process time series;

[0034] The third layer is the random dropout layer: in each training batch, by ignoring a predetermined number of feature detectors, the overfitting phenomenon can be significantly reduced. During forward propagation, the activation values ​​of the neurons are kept at a preset node retention probability. Stop working, which can make the model more generalizable because it will not rely too much on certain local features;

[0035] The fourth and fifth layers are fully connected layers: they map the time series processed by the long short-term memory network layer to the sample tag space, and finally output the prediction result in the dimension of the state value;

[0036] The sixth layer is the regression output layer: the predicted result, i.e. the output of the long short-term memory network, is compared with the actual value of the power grid in the preprocessed data, and the loss value is calculated. The loss value is used to update the weights and biases of the network during the back propagation process.

[0037] Preferably, the LSTM network has a corresponding unit state at each moment, and the unit state is similar to a conveyor belt, which runs directly on the entire chain with only a small amount of linear interaction. It is easy for information to flow on it and remain unchanged, and the ability to remove or add information to the unit state is achieved through a carefully designed structure called a "gate";

[0038] In the forward propagation, the LSTM network uses the forget gate and the input gate to control the cell state. The forget gate determines how much of the cell state at the previous moment is retained to the current moment. ; The input gate determines the input state vector of the network at the current moment How much is saved to the cell state In addition, the LSTM network uses output gates to control the cell state. How many outputs are there to the current output value of the LSTM network? , the specific steps are as follows:

[0039] Forget gate: determines what information to discard from the cell state. This gate reads the output value of the LSTM network at the previous moment. and the input state vector of the network at the current moment , output a value between 0 and 1 corresponding to the unit state at the previous moment For each value in , 1 means "completely retained" and 0 means "completely discarded". The formula is as follows:

[0040] (7)

[0041] In formula (7), is the output of the forget gate, is the Sigmoid activation function, is the weight matrix of the forget gate, is the output value of the LSTM network at the previous moment, is the input state vector of the network at the current moment is the activation value of the forget gate;

[0042] Input gate: determines which new information is stored in the cell state, including sigmoid layer and tanh layer: the sigmoid layer determines which values ​​will be updated; the tanh layer creates a new candidate value vector , and added to the state, the formula is as follows:

[0043] (8)

[0044] In formula (8), and is the output of the input gate, is the Sigmoid activation function, is the tanh activation function, and is the weight matrix of the input gate, and is the activation value of the input gate;

[0045] Further, update the old unit state, that is Updated to : By combining the old cell state with Multiply, discard the information that needs to be discarded, and then add , which is the new candidate value, changes according to the degree of updating each state, and the formula is as follows:

[0046] (9)

[0047] In formula (9), is the unit state at the current moment, is the output of the forget gate, is the unit state at the previous moment, and is the output of the input gate;

[0048] Output gate: This determines the output value, which is based on the cell state, but is also a filtered version. First, a sigmoid layer is run to determine which part of the cell state will be output. Then, the cell state is processed through tanh to obtain a value between -1 and 1, and this value is multiplied by the output of the sigmoid gate to finally output the part that determines the output. The formula is as follows:

[0049] (10)

[0050] In formula (10), and is the output of the output gate, is the Sigmoid activation function, is the tanh activation function, is the weight matrix of the output gate, is the activation value of the output gate;

[0051] In backpropagation, the error term is passed backward through time and the state of each cell is calculated The error terms of the outputs of the three gates, that is, starting from the current moment, the error terms are calculated at each moment, and then the error terms are propagated to the upper layer and used to calculate the error terms of the nodes in the upper layer; when calculating the gradient of a node, first find the output node of the node, then calculate the gradient of all output nodes and multiply it by the gradient of the output node to the node, and finally add them up to get the gradient of the node;

[0052] According to the corresponding error term, the gradient of each weight is calculated and the internal parameters of the network are updated.

[0053] According to the preferred embodiment of the present invention, the partially observable Markov decision problem in step S3 is firstly given an agent and an environment, and the discrete-time Markov observation is composed of seven tuples: Definition, where represents the set of hidden states of the environment, Represents the action set of the terminal, represents the conditional transition probability set between states, : S×A→R represents the reward function that maps state-action pairs to rewards, represents the observation set of the terminal, represents the set of conditional observation probabilities, and is the discount rate, which indicates the degree to which current rewards are preferred over future rewards;

[0054] At every moment , the environment is hidden , with probability Get the right Observation, the terminal takes action , and receives rewards from the environment based on its actions and the current state of the environment , and at the same time, the environment is probabilistic Transition to next state , repeat the process until the terminal state is reached; in this process, the terminal goal is to determine an optimal strategy , which maps observations to actions and maximizes the agent’s expected total discounted return, i.e. ; Similarly, if the terminal obtains costs rather than rewards from the environment, the goal is to minimize the expected total discounted cost, so the partially observable Markov decision problem can be written as follows:

[0055] (11)

[0056] In formula (11), represents the optimal strategy, represents the observation set of the terminal, represents the action set of the terminal, t represents the time, represents the discount rate, Indicates reward;

[0057] According to the preferred embodiment of the present invention, the problem of using the estimated power grid state to determine whether there is an attack is modeled as a partially observable Markov decision problem setting: Assume that at an unknown time τ, a network attack is launched on the system. Due to the unknown attack launch time τ, there are two hidden states : Before and after the attack; at each time t, basic observations of the system and the measurement vector are available , based on the analysis of the processing of the observed system, the basic state of the system can be determined and two actions can be taken : Stop and declare the attack and continue to make further measurements. If the basic state is before the attack, continue to make further measurements. If the basic state is after the attack, stop and declare the attack exists. The algorithm for judging the basic state through basic observation and taking action is described in S4.

[0058] Since the detection goal is to detect an attack as soon as possible after it occurs, that is, to minimize detection delay and false alarm rate, false alarm and detection delay events are associated with certain costs. Assume that the relative cost of detection delay compared to false alarm events is ; then, if the true base state is before attack and the action stop is chosen, a false alarm occurs and the defender receives a cost of 1, and if the base state is after attack and the action continue is chosen, the defender receives a cost of 1 due to the detection delay , for all other hidden state-action pairs, the cost is assumed to be zero; furthermore, once an action stop is chosen, no further cost is incurred while staying in the terminal state, with the ultimate goal being to minimize the expected total cost by choosing one's actions appropriately.

[0059] Preferably, according to the present invention, step S4 includes the following specific steps:

[0060] S41, residuals of system states estimated by LSTM and measured values As observation data, and quantify the observation data;

[0061] Since the observation space is continuous, mapping from every possible observation to action is computationally infeasible. To reduce the computational complexity of such a continuous space, the observation data can be quantized. Using a quantization threshold Divide the observation space into mutually exclusive and non-intersecting intervals, when hour, is the residual of the estimated system state and the measured value, that is, the observation before quantization at time t, and the observation after quantization at time t is It means that each observation are assigned different values, and observations at different given times are ;

[0062] The residual The calculation formula is as follows:

[0063] (12)

[0064] In formula (12), For in time The measurement vector at is the measurement matrix determined based on the network topology, is the long short-term memory network for time The estimate of the state vector at , T is the transpose;

[0065] S42. The observation signal should be informative to reduce the uncertainty about the underlying state. Since different states require different optimal actions, additional information from the observation history should be used to further reduce the ambiguity about the underlying state: a finite-sized sliding observation window is used as memory, that is, a finite number of historical observations are recorded as a history window, and the most recently recorded history window is mapped to an action. The size of the sliding observation window is ,exist observation window, at time The sliding window is composed of The quantized version of

[0066] S43. During the reinforcement learning process, a Q-value table is continuously maintained to determine which action should be taken in the current observation window, that is, it is necessary to learn the direct mapping from observation to action, that is, to determine whether the basic state is before attack or after attack through observation, and take action according to the basic state: if the basic state is before attack, the operation is to continue; if the basic state is after attack, the operation is to stop; the Q-value table is continuously iterated to make it converge, and then an optimal strategy is selected in each state according to the Q-value table;

[0067] The dimension of the Q value table is: number of states * number of actions. Each number in the table represents the Next action The discounted sum of future earnings.

[0068] Preferably, according to the present invention, the training algorithm flow of the Q value table maintained in the reinforcement learning process is:

[0069] First, initialize the Q value table and construct a Q value table with m rows and n columns, where n is the number of operations and m is the number of states, and initialize the value to 0;

[0070] Then, select and execute the operation, and keep running this step until you stop training or the training loop stops;

[0071] Finally, evaluate the actions taken and observe the results and rewards, updating the function :

[0072] (13)

[0073] In formula (13), is the updated Q value, The Q-value of taking action a for state s, is the learning rate, is the reward for taking action a in state s, An estimate of future rewards, where is a discount factor, usually less than 1, used to reduce the impact of future rewards on the current Q value. Indicates that in the next state All possible actions The maximum value of the Q value, which represents the maximum cumulative reward that can be obtained in the future starting from the current action;

[0074] According to the trained Q-value table, the current observation window can be mapped to the best action during attack detection to determine whether an attack exists; the algorithm is universal, that is, it does not require an attack model. This makes the proposed scheme widely applicable and proactive, and can detect new unknown attack types.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) The long short-term memory network (LSTM) with strong nonlinear fitting ability is used to estimate the state of the power system. It can not only accurately capture the complex nonlinear relationship between the measured data and the real-time state of the power grid, thereby significantly improving the estimation accuracy of the system state, but also uniquely retain the critical long-term dependency information in time series data.

[0077] (2) The sliding window technology is introduced to fully capture the continuous dynamic change characteristics contained in the power grid state. This technology effectively reduces the uncertainty and ambiguity in the judgment process of the basic power grid state by integrating and analyzing the additional information in the observation history. Subsequently, reinforcement learning is used to map this window containing rich historical information to a specific operation or decision-making problem, thereby achieving effective detection of hidden attacks in the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a schematic diagram of the long short-term memory network of the present invention.

[0079] Figure 2 It is a diagram showing the principle of reinforcement learning of the present invention.

[0080] Figure 3 It is a state estimation diagram based on the long short-term memory network of the present invention.

[0081] Figure 4 It is the impact diagram of the attack on the state estimation used in the present invention.

[0082] Figure 5 The diagram is a schematic diagram of the effect of the attack detection based on reinforcement learning of the present invention under various attacks, wherein (a) is a schematic diagram of the effect of the attack detection based on reinforcement learning under a mixed attack of false data injection and noise interference, (b) is a schematic diagram of the effect of the attack detection based on reinforcement learning under a false data injection attack, and (c) is a schematic diagram of the effect of the attack detection based on reinforcement learning under a denial of service attack. DETAILED DESCRIPTION

[0083] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0084] Embodiment 1,

[0085] A time series prediction reinforcement learning detection method for covert attacks on smart grids, the method comprising:

[0086] S1. Model the smart grid, obtain instrument measurement data, and pre-process the data;

[0087] S2, using the preprocessed data as the input of the long short-term memory network to perform state estimation on the smart grid;

[0088] S3,models the attack detection problem in smart grid as a partially observable Markov decision problem;

[0089] S4. Solve the partially observable Markov decision problem based on reinforcement learning and realize the detection of hidden attacks on smart grids.

[0090] (1) Modeling the smart grid

[0091] Smart grid is based on physical grid and integrates modern sensor measurement technology, communication technology, information technology, computer technology and control technology. Smart grid covers all links from power generation, transmission, distribution to power consumption, and uses smart bus as the key connection point to achieve efficient transmission and flexible dispatch of electric energy. The power grid consists of busbars. instruments, of which usually , to achieve the necessary measurement redundancy against noise. Take one of the buses as the reference bus and use To represent the smart grid The state vector at time t, where represents the state value at bus n at time t. The measured value at The measurement vector is represented by Based on the widely used nonlinear DC power system model, the smart grid is modeled with the following state space equations:

[0092]

[0093]

[0094] in, is the system (state transition) matrix, It is a measurement matrix determined based on the network topology. is the process noise vector, is the measurement noise vector.

[0095] (2) Smart grid state estimation based on long short-term memory network

[0096] Since smart grids are regulated based on estimated system states, state estimation is a basic task in smart grids, and static least squares estimators are usually used. However, in practice, the bidirectional time-varying of load and generation makes the operation state of smart grids highly dynamic. Long short-term memory networks can not only finely capture the complex nonlinear associations between measurement data and real-time states, but also preserve the long-term dependencies of time series, and better estimate the system state. By estimating the current network state through the long short-term memory network, the long-term dependencies in the smart grid state are retained with the help of its gating and unit state mechanism, and the problems of gradient vanishing and gradient exploding will not occur, thereby improving the more detailed observation of the smart grid state and improving the performance of attack detection.

[0097] The data obtained by the measuring instrument can be used as the input of the network for learning and use only after preprocessing. The specific network contains the following six layers of structure.

[0098] The first layer is the sequence input layer: the input data is converted into a sequence format and passed to the next layer for processing. The feature dimension of the input sequence is equal to the dimension of the measurement value.

[0099] The second layer is the long short-term memory network layer: the core layer of the model, which processes time series. Modeling is done by learning the temporal patterns and dependencies in the input sequence, and adjusting the model parameters through the back propagation algorithm to minimize the error between the prediction and the actual value. Through continuous iterative training and optimization, the long short-term memory network model gradually improves its prediction performance, so that it can better process time series.

[0100] The third layer is a random dropout layer: in each training batch, by ignoring a predetermined number of feature detectors, overfitting can be significantly reduced. During forward propagation, the activation value of a neuron is set to a predetermined probability. Stop working, which makes the model more generalizable because it will not rely too much on certain local features.

[0101] The fourth and fifth layers are fully connected layers: the learned feature representations are mapped to the sample label space, and the final output dimension is the dimension of the state value.

[0102] The sixth layer is the regression output layer: the predicted result is compared with the actual value and the loss value is calculated. In the back propagation process, the loss value will be used to update the weights and biases of the network.

[0103] LSTM networks have a cell state at each moment in time. The cell state is like a conveyor belt, running directly across the chain with only a few linear interactions. It is easy for information to flow through it unchanged, with the ability to remove or add information to the cell state through carefully designed structures called "gates".

[0104] In the forward propagation, the LSTM network uses two gates to control the cell state. The first is the forget gate, which determines how much of the cell state at the previous moment is retained at the current moment. ; The other is the input gate, which determines the input of the network at the current moment How much is saved to the cell state LSTM networks use output gates to control cell states. How many outputs are there to the current output value of the LSTM network? The principle is as follows Figure 1 As shown, the specific steps are as follows:

[0105] Forget gate: determines what information to discard from the cell state. This gate reads and , outputs a value between 0 and 1 for each cell state The value in . 1 means "completely retain" and 0 means "completely discard", which can be expressed by the following formula:

[0106]

[0107] Input gate: determines what new information is stored in the cell state. There are two parts here. First, a sigmoid layer determines which values ​​will be updated. Then, a tanh layer creates a new candidate value vector , and added to the state, it can be expressed by the following formula:

[0108]

[0109] Update the old cell state, i.e. Updated to By combining the old state with Multiply, discard the information that needs to be discarded, and then add , which is the new candidate value. The degree of updating each state varies according to the decision, which can be expressed by the following formula:

[0110]

[0111] Output gate: This determines the output value. This output is based on the cell state, but it is also a filtered version. First, a sigmoid layer is run to determine which part of the cell state will be output. Next, the cell state is processed through tanh (to get a value between -1 and 1) and multiplied by the output of the sigmid gate. Finally, only the part that determines the output is output, which can be expressed as follows:

[0112]

[0113] In the reverse propagation, the error term is transmitted in the reverse direction of time, and the error term of each neuron is calculated. That is, starting from the current moment, the error term is calculated at each moment, and then the error term is propagated to the upper layer and used to calculate the error term of the nodes in the upper layer. When calculating the gradient of a node, first find the output node of the node, then calculate the gradient of all output nodes and multiply it by the gradient of the output node to the node, and finally add them up to get the gradient of the node.

[0114] According to the corresponding error term, the gradient of each weight is calculated and the internal parameters of the network are updated.

[0115] (3) Modeling the attack detection problem in smart grids as a partially observable Markov decision problem

[0116] Covert attacks can bypass the system's anomaly detection mechanism and cause damage to the system. They have multiple modes and strong system adaptability, and are a risk to the safe operation of smart grid systems that can pose a serious threat. Traditional bad data detection technology relies on linear estimators and residual thresholds to detect malicious attacks, and uses the last moment's observation data to estimate the operating state, which is difficult to deal with covert attacks. In order to effectively detect covert attacks in power grid systems, it is necessary to use the system state estimated by long short-term memory networks to determine whether the current system is attacked: in the state before the attack, the system operates under normal conditions, and using the system model, normal measurements can be obtained with high precision. After the attack, the measurements will change into different unknown forms according to the attacker's strategy. Considering the different system states before and after the attack, there is a hidden state of an unknown change point, and considering the fastest detection of the change after the attack, the attack detection problem can be further described as a partially observable Markov decision problem.

[0117] For partially observable Markov decision problems, given an agent and an environment, the discrete-time Markov observations consist of a seven-tuple Definition, where represents the set of (hidden) states of the environment, Represents the action set of the terminal, represents the conditional transition probability set between states, : S×A→R represents the reward function that maps state-action pairs to rewards, represents the observation set of the terminal, represents the set of conditional observation probabilities, and is the discount rate, which indicates the degree to which current rewards are preferred over future rewards.

[0118] At every moment , the environment is hidden With probability Get the right Observation, the terminal takes action , and receives rewards from the environment based on its actions and the current state of the environment At the same time, the environment is based on probability Transition to next state The process is repeated until the terminal state is reached. In this process, the terminal goal is to determine an optimal strategy , which maps observations to actions and maximizes the agent’s expected total discounted return, i.e. . Similarly, if the terminal obtains costs from the environment instead of rewards, the goal is to minimize the expected total discounted cost. Therefore, the partially observable Markov decision problem can be written as follows:

[0119]

[0120] Since the goal is to minimize detection delay and false alarm rate, both false alarm and detection delay events should be associated with a certain cost. Assume that the relative cost of detection delay compared to false alarm events is Then, if the true underlying state is before attack and the action stop is selected, a false alarm occurs and the defender receives a cost of 1. On the other hand, if the underlying state is after attack and the action continue is selected, the defender receives a cost of 1 due to the detection delay. For all other (hidden) state-action pairs, the cost is assumed to be zero. Furthermore, once an action stop is chosen, no further cost is incurred for staying in the terminal state, with the ultimate goal being to minimize the expected total cost by choosing one's actions appropriately.

[0121] (4) Detection of hidden attacks on smart grids based on reinforcement learning

[0122] The problem solved by the reinforcement learning (RL) algorithm is how an intelligent agent can maximize the rewards it can obtain in a complex and uncertain environment. By perceiving the reaction of the state of the environment to the action, it can guide better actions and obtain the maximum benefit. It is therefore considered to be an effective method to control uncertain environments. Therefore, RL can be used to effectively solve the partially observable Markov decision problem described. Due to the unknown attack launch time and unknown attack strategies, and known reinforcement algorithms are effective in uncertain environments. In addition, since model-based methods require a more computationally demanding two-step solution and can usually only learn an approximate model, a model-free reinforcement method is used to solve partially observable Markov decision problems.

[0123] Since the observation space is continuous, mapping from every possible observation to action is computationally infeasible. To reduce the computational complexity of such a continuous space, the observation data can be quantized. Using a quantization threshold Divide the observation space into mutually exclusive and disjoint intervals, so that if Then the observation at time t is Then, the possible observations at any given time are .because is the representation of the quantization level, so each Need to be assigned different values.

[0124] In order to determine the best action, observations need to be used to infer the underlying state, and the observation signal should be informative to reduce the uncertainty about the underlying state. Since different states require different optimal actions, additional information from the observation history should be used to further reduce the ambiguity about the underlying state. A finite-sized sliding observation window is used as memory, and the most recent history window is mapped to an action. Assume that the size of the sliding observation window is , so that there exists possible observation windows, and at time The sliding window is composed of After that, the observation window is passed so that the observation space O consists of all possible observation windows.

[0125] In the process of reinforcement learning, the agent is constantly interacting with the environment. Figure 2 As shown in the figure, the agent obtains the state in the environment, and uses this state to output an action, a decision. Then this decision will be put into the environment, and the environment will output the next state and the reward for the current decision based on the decision taken by the agent. The purpose of the agent is to obtain as many rewards as possible from the environment.

[0126] The specific algorithm is to continuously maintain a Q value table during the learning process to determine which action should be taken in the current observation window. That is, it is necessary to learn the direct mapping from observation to action, that is, the stop time. If the basic state is before the attack, the best action is to continue; if the basic state is after the attack, the best action is to stop. The dimension of the table is: number of states * number of actions, each number in the table represents the current state The following actions can be taken The discounted sum of future benefits that can be obtained. The Q-value table is continuously iterated to make it converge, and then an optimal strategy can be selected in each state based on the Q-value table.

[0127] The training algorithm process is:

[0128] Step 1: Initialize the Q value table. First, construct a Q value table with m rows and n columns. n is the number of operations and m is the number of states. And initialize the value to 0.

[0129] Steps 2 and 3: Select and execute an action. The combination of these steps is completed in an indeterminate amount of time. This means that this step runs until you stop training, or the training loop stops.

[0130] Steps 4 and 5: Evaluate the actions taken and observe the results and rewards, update the function :

[0131]

[0132] According to the trained Q-value table, the current observation window can be mapped to the best action during attack detection to determine whether an attack exists. The algorithm is universal, that is, it does not require an attack model. This makes the proposed scheme widely applicable and proactive, and can detect new unknown attack types.

[0133] Embodiment 2,

[0134] Experiment: In a Node and The experiment was conducted on the IEEE-14 node power system data composed of 10 smart meters. Taking the IEEE-14 power system as an example, the system matrix As the identity matrix, determine the measurement matrix .

[0135] For the state estimation scheme based on long short-term memory network, the training batch is 300 times, the input data is the regularized measurement value, and the output is the regularized state value. In the test for a period of time, the average absolute error is 0.019, and the average root mean square error is 0.0241. Taking the first node as an example, its true value and estimated value are as follows Figure 3The state estimation diagram based on the long short-term memory network is shown in the figure. The horizontal axis is time, and the vertical axis is the true value and the estimated value. Between the time points 100 and 150, after injecting an attack with a size of about 1% of the measured value, the impact on the state estimation is as follows Figure 4 As shown, the horizontal axis is time, and the vertical axis is the estimated value of the state with and without attack.

[0136] For the proposed enhanced stealth attack detection scheme, the quantization level is selected , the number of sliding observation windows In addition, the learning parameters are chosen as During the learning phase, the defender was first trained over 20,000 times with an attack time of 100, and then further trained over 20,000 times with an attack time of 1, to ensure that the defender fully explored the observation space under normal operating conditions as well as under attack conditions.

[0137] The false positive rate and average detection delay are used as the performance indicators of attack detection. Both performance indicators depend on the attack time. Therefore, for each possible Calculate the performance index, choose τ as a geometric random variable with parameter ρ, so that The effectiveness of the present invention is tested using three attack modes: false data injection attack, false data injection and noise interference mixed attack, and denial of service attack.

[0138] False data injection attack is an attack method that exploits the bad data detection loopholes in the power system to inject designed attack vectors into the measurement vectors, thereby maliciously tampering with the state estimation results, seriously endangering the safe and reliable operation of the power system.

[0139] The hybrid attack of false data injection and noise interference is to add additional noise to interfere with the state estimation results based on the false data injection attack.

[0140] Denial of service attacks usually exploit transmission protocol weaknesses, system vulnerabilities, and service vulnerabilities to attack the target system, making some instrument measurements in the system unavailable.

[0141] The experimental results are as follows Figure 5 As shown, Figure 5 (a) shows the attack detection test under different attack times under the mixed attack of false data injection and noise interference. Figure 5 (b) is the experimental result diagram under false data injection attack. Figure 5Figure (c) in Figure 1 is the experimental result diagram under denial of service attack. Under different attack densities, the false detection rate is inversely proportional to the average detection delay. In general, it can be seen that the present invention can detect hidden attacks in smart grids with lower delay and false detection rate.

Claims

1. A time series prediction reinforcement learning detection method for covert attacks on smart grids, characterized in that: The method comprises: S1. Model the smart grid, obtain instrument measurement data, and pre-process the data; S2, using the preprocessed data as the input of the long short-term memory network to estimate the state of the smart grid and determine whether there is an attack; The long short-term memory network includes the following six layers: The first layer is the sequence input layer: the instrument measurement values ​​in the preprocessed data are converted into sequence format as input data and passed to the next layer for processing. The feature dimension of the input sequence is equal to the dimension of the measurement value. The second layer is the long short-term memory network layer: it models by learning the temporal patterns and dependencies in the input sequence, and adjusts the long short-term memory network model parameters through the back-propagation algorithm to minimize the error between the prediction and the actual value; The third layer is a random dropout layer: it ignores a predetermined number of feature detectors: during forward propagation, the activation values ​​of neurons are kept at a preset node retention probability. Stop working; The fourth and fifth layers are fully connected layers: the sequence processed by the long short-term memory network layer is mapped to the sample tag space, and the dimension of the final output prediction result is the dimension of the state value; The sixth layer is the regression output layer: the prediction result, i.e. the output of the LSTM network, is compared with the actual value of the power grid in the preprocessed data, and the loss value is calculated, where the loss value is used to update the weights and biases of the network in the back-propagation problem; S3, the problem of using the estimated power grid state to determine whether there is an attack is modeled as a partially observable Markov decision problem; S4. Solve the partially observable Markov decision problem based on reinforcement learning and realize the detection of hidden attacks on smart grids.

2. The method for detecting hidden attacks on smart grids by time series prediction and reinforcement learning according to claim 1 is characterized in that: In S1, the smart grid is modeled and the instrument measurement data is obtained, which means: the smart grid includes Busbars and Instruments, including , taking one of the busbars as the reference busbar, Indicates that smart grid The state vector at the moment is as follows: （1） In formula (1), represents the state value at bus n at time t; T is transpose; Smart grid The measurement vector at the moment is as follows: （2） In formula (2), Indicates at time The measured value at the instrument k; T is the transpose; Based on the widely used nonlinear DC power system model, the smart grid is modeled with the following state-space equations: （3） （4） In formula (3) and (4), is the system matrix, is a measurement matrix determined based on the network topology, is the problem noise vector, is the measurement noise vector.

3. The time series prediction reinforcement learning detection method for covert attacks on smart grids according to claim 2 is characterized in that: The preprocessed data includes instrument measurement values and actual value of the power grid The preprocessing process is: obtain T instrument measurement values ​​in continuous time and calculate their mean vector and the standard deviation vector , then at time The measurement vector at After processing, it is expressed as （5）； Get the actual values ​​of T power grids in continuous time and calculate their mean vector and the standard deviation vector , then at time The state vector at After processing, it is expressed as （6）。 4. The method for detecting hidden attacks on smart grids by using time series prediction reinforcement learning according to claim 1 is characterized in that: The long short-term memory network has a corresponding unit state at each moment ; In the forward propagation, the LSTM network uses the forget gate and the input gate to control the cell state The forget gate determines how much of the cell state at the previous moment is retained to the current moment. ; The input gate determines the input state vector of the network at the current moment How much is saved to the cell state In addition, the LSTM network uses output gates to control the cell state. How many outputs are there to the current output value of the LSTM network? , the specific steps are as follows: Forget gate: read the output value of the long short-term memory network at the previous moment and the input state vector of the network at the current moment , output a value between 0 and 1 corresponding to the unit state at the previous moment For each value in , 1 means "completely retain" and 0 means "completely discard". The formula is as follows: （7） In formula (7), is the output of the forget gate, is the Sigmoid activation function, is the weight matrix of the forget gate, is the output value of the LSTM network at the previous moment, is the input state vector of the network at the current moment is the activation value of the forget gate; Input gate: The input gate includes a sigmoid layer and a tanh layer: the sigmoid layer determines which values ​​will be updated; the tanh layer creates a new candidate value vector , and added to the state, the formula is as follows: （8） In formula (8), is the output of the input gate, is the Sigmoid activation function, is the tanh activation function, and is the weight matrix of the input gate, is the activation value of the input gate; Further, update the old unit state, that is Updated to : Combine the old unit state with Multiply, discard the information that needs to be discarded, and then add , forming new candidate values, which change according to the degree of decision to update each state. The formula is as follows: （9） In formula (9), is the unit state at the current moment, is the output of the forget gate, is the unit state at the previous moment, and is the output of the input gate; Output gate: First, run a sigmoid layer to determine which part of the cell state is output; then, process the cell state through tanh to get a value between -1 and 1, and multiply this value with the output of the sigmoid gate to finally output the part that determines the output. The formula is as follows: （10） In formula (10), and is the output of the output gate, is the Sigmoid activation function, is the tanh activation function, is the weight matrix of the output gate, is the activation value of the output gate; In backpropagation, the error term is passed backward through time and the state of each cell is calculated The error terms of the outputs of the three gates, that is, starting from the current moment, the error terms are calculated at each moment, and then the error terms are propagated to the upper layer for the nodes in the upper layer to calculate the error terms; When calculating the gradient of a node, first find the output node of the node, then calculate the gradient of all output nodes and multiply it by the gradient of the output node to the node, and finally add them together to get the gradient of the node; According to the corresponding error term, the gradient of each weight is calculated and the internal parameters of the network are updated.

5. The method for detecting hidden attacks on smart grids by time series prediction and reinforcement learning according to claim 1 is characterized in that: The partially observable Markov decision process in step S3 is first given an agent and an environment, and the discrete-time Markov observation is composed of seven tuples. Definition, where represents the set of hidden states of the environment, Represents the action set of the terminal, represents the conditional transition probability set between states, : S×A→R represents the reward function that maps state-action pairs to rewards, represents the observation set of the terminal, represents the set of conditional observation probabilities, and is the discount rate, which indicates the degree to which current rewards are preferred over future rewards; At every moment , the environment is hidden , with probability Get the right Observation, the terminal takes action , and receives rewards from the environment based on its actions and the current state of the environment , and at the same time, the environment is probabilistic Transition to next state , repeat the problem until a terminal state is reached; in this problem, the terminal goal is to determine an optimal policy , which maps observations to actions and maximizes the agent’s expected total discounted return, i.e. ; Similarly, if the terminal obtains costs rather than rewards from the environment, the goal is to minimize the expected total discounted cost, so the partially observable Markov decision problem can be written as follows: （11） In formula (11), represents the optimal strategy, represents the observation set of the terminal, represents the action set of the terminal, t represents the time, represents the discount rate, Indicates reward.

6. The method for detecting hidden attacks on smart grids by using time series prediction reinforcement learning according to claim 5 is characterized in that: The problem of using the estimated power grid state to determine whether there is an attack is modeled as a partially observable Markov decision problem setting: at an unknown time τ, a network attack is launched on the system. Due to the unknown attack launch time τ, there are two hidden states : Before and after the attack; at each time, basic observations of the system and measurement vectors are available , based on the analysis of the processing of the observed system, the basic state of the system can be determined and two actions can be taken : Stop and declare the attack and continue with further measurements. If the base state is before the attack, continue with further measurements. If the base state is after the attack, stop and declare the attack exists. The relative cost of a detection delay compared to a false alarm event is ; then, if the true base state is before attack and the action stop is chosen, a false alarm occurs and the defender receives a cost of 1, and if the base state is after attack and the action continue is chosen, the defender receives a cost of 1 due to the detection delay , for all other hidden state-action pairs, the cost is set to zero; furthermore, once an action stop is chosen, no further cost is incurred while staying in the terminal state, with the ultimate goal of minimizing the expected total cost by choosing one's actions appropriately.

7. The method for detecting hidden attacks on smart grids by using time series prediction reinforcement learning according to claim 1 is characterized in that: The step S4 comprises the following specific steps: S41, residuals of system states estimated by LSTM and measured values As observation data, and quantify the observation data; use the quantization threshold Divide the observation space into mutually exclusive and non-intersecting intervals, when hour, is the residual of the estimated system state and the measured value, that is, the observation before quantization at time t, and the observation after quantization at time t is It means that each observation are assigned different values, and observations at different given times are ; The residual The calculation formula is as follows: （12） In formula (12), For in time The measurement vector at is the measurement matrix determined based on the network topology, is the long short-term memory network for time The estimate of the state vector at , T is the transpose; S42. Use additional information from the observation history to further reduce the ambiguity about the underlying state: Use a finite-sized sliding observation window as memory, that is, record a finite number of historical observations as a history window, and map the most recently recorded history window to an operation. The size of the sliding observation window is ,exist observation window, at time The sliding window is composed of The quantized version of S43. In the reinforcement learning problem, a Q-value table is continuously maintained to determine which action should be taken in the current observation window, that is, it is necessary to learn the direct mapping from observation to action, that is, to determine whether the basic state is before attack or after attack through observation, and take action according to the basic state: if the basic state is before attack, the operation is to continue; if the basic state is after attack, the operation is to stop; the Q-value table is continuously iterated to make it converge, and then an optimal strategy is selected in each state according to the Q-value table; The dimension of the Q value table is: number of states * number of actions. Each number in the table represents the Next action The discounted sum of future earnings.

8. The method for detecting hidden attacks on smart grids by using time series prediction reinforcement learning according to claim 5 is characterized in that: The training algorithm flow of the Q value table maintained in the reinforcement learning problem is: First, initialize the Q value table and construct a Q value table with m rows and n columns, where n is the number of operations and m is the number of states, and initialize the value to 0; Then, select and execute the operation, and keep running this step until you stop training or the training loop stops; Finally, evaluate the actions taken and observe the results and rewards, updating the function : （13） In formula (13), is the updated Q value, The Q-value of taking action a for state s, is the learning rate, is the reward for taking action a in state s, An estimate of future rewards, where is a discount factor, less than 1, used to reduce the impact of future rewards on the current Q value. Indicates that in the next state All possible actions The maximum value of the Q value, which represents the maximum cumulative reward that will be obtained in the future starting from the current action; According to the trained Q-value table, the current observation window is mapped to the best action during attack detection to determine whether there is an attack.

Images