Security-aware approach for vulnerability attack path concealment based on power network

Abstract

The application relates to the technical field of power networks, and provides a security perception method based on a power network and capable of being used for hiding a vulnerability attack path, which comprises the following steps: 1) modeling a Markov decision process (MDP) with security constraints; 2) modeling linear time logic (LTL) specifications; 3) modeling an attacker attack model and initial state opacity; 4) modeling a reinforcement learning problem under security constraints; 5) constructing an initial state estimator (ISE) of the MDP and building a task model; and 6) solving by using Q learning to obtain an optimal strategy, namely completing security perception. The security perception method can comprehensively consider the security requirement of power network routing, system behavior specifications, initial state opacity and potential intruder attacks, so that an effective reinforcement learning scheme is provided to optimize a routing strategy and guarantee the security of the power network when messages are sent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power network, in particular, to a security-aware method based on power network for hiding vulnerability attack path. BACKGROUND

[0002] Network security of power industry control system is an important issue. The control system of power enterprise contains rich information, and the interest value behind these information is very high. Once leaked, it will not only affect personal safety, but also seriously affect national security. In addition, the power enterprise is directly related to people's livelihood. If information leakage occurs, it will cause certain panic to the society. In addition, the attack of hostile forces is also a factor threatening the security of power grid. In recent years, the number of attacks on power grid by hostile forces has been rising. They not only start from the accuracy and direction of the attack, but also constantly change the source of the attack. In addition, there are also problems of power industry control system itself. In view of the distribution range of power grid and the complexity of power grid distribution, it is easy to cause the problem of reducing safety factor due to improper management or unclear organization. At present, the power grid system is more closely related to the computer system, and the security requirement of power grid security to the computer system is higher. Therefore, it is of great significance to protect the network security of power system. In the face of strong uncertainty of new power system, rapid growth of control scale, low-carbon operation target and other changes, the power network presents high-dimensional, nonlinear and non-convex complex characteristics. Data-driven algorithms represented by reinforcement learning bring opportunities to solve network optimization problems in the field of power grid. The routing of power network may leak the node information of network and threaten the network security, so a security-aware method based on power network for hiding vulnerability attack path is needed. SUMMARY

[0003] The present application provides a security-aware method based on power network for hiding vulnerability attack path, which can solve the problem of ensuring the security of the initial area not being leaked when sending messages of power network routing.

[0004] The security-aware method based on power network for hiding vulnerability attack path according to the present application comprises the following steps:

[0005] 1) modeling Markov decision process (MDP) with security constraints;

[0006] 2) modeling linear time logic (LTL) specification;

[0007] 3) modeling invader attack model and opacity of initial state;

[0008] 4) modeling reinforcement learning problem under security constraints;

[0009] 5) constructing an initial state estimator ISE of the MDP, building a task model;

[0010] 6) solving with Q-learning to get an optimal policy, i.e. completing the safety perception.

[0011] As preferred, in step 1), one MDP is the environment faced by the agent, which is modeled as:

[0012] M = (X, A, X0, f, AP, L), where M denotes the MDP Markov Decision Process model, X is a finite set of states; A is a finite set of actions; X0is a finite set of initial states, i.e. the initial state of the agent is a subset of all possible states; f: X x A x X -> [0, 1] is a state transition function; L is a labeling function;

[0013] Given a MDP M, a path x0∈X0is a sequence

[0014]

[0015] where x[i] denotes the state of the agent at time i, x[1] = x0, i.e. the agent is in the initial state x0at the first time; a policy π satisfies f(x[i], π(x[1]a[1]...x[i]), x[i+1]) > 0, where π(x[1]a[1]...x[i]) denotes a series of policies taken by the agent, i.e. taking action a[1] policy at state x[1], taking action a[2] policy at state x[2]... until moving to state x[i]; f(x[i], π(x[1]a[1]...x[i]), x[i+1]) x[i+1]) denotes a state transition probability, i.e. the probability of the agent moving to x[i+1] after taking policy π(x[1]a[1]...x[i]) at state x[i]; X ω is the set of all infinite sequences over X.

[0016] As preferred, in step 2), LTL formulas φ over the set of atomic propositions AP are formed according to the following syntax:

[0017]

[0018] where φ denotes a logical task model, a is an atomic proposition, φ1and φ2denote sub-logical task models, ○ represents next, U represents until, and true denotes true result;

[0019] For a MDP M, a trace is a sequence:

[0020] t = H(p) = H(x[0]) H(x[1]) H(x[2])... e H(x) ω such that f(x[i], π(x[1] a[1]... x[i]), x[i+1]) > 0 for all i, where H is a labeling function representing the state of the agent as seen by the observer, e.g. H(x[1]) represents the state of the agent as seen by the observer in state x[1], H(p) represents the trajectory of the agent as seen by the observer from state x[0] to the end, and ω represents an infinite sequence. As a preference, in step 3), the initial state opacity is modeled by taking security requirements, given a MDP M = (X, A, X0, f, AP, L), assume there is a set of secret states X0 represents all possible states, Xs represents all possible initial states, an intruder can observe the output sequence of the system, which is generated by the output function H: X → Y, where Y is the output set, H can be extended to be generated by H(p) = H(x[1]) H(x[2])... ;

[0021] Under the policy π, the initial opaque state with respect to x0 e XS is defined as:

[0022] such that H(s) = H(t), i.e. for a path 1 consisting of a sequence of states starting from x[1] = x0, if there exists a path 2 starting from some state in the initial state set Xs, which is different from path 1, i.e. each state x'[i] in path 2 has a chance to be different from the state x[i] in path 1, in theory the agent has a certain probability to walk path 2, and the observer sees the same state at each time of path 1 and path 2, then the initial state x0 is opaque.

[0023] As a preference, in step 4), the modeling of the reinforcement learning problem under security constraints: given a MDP M and secret states an initial state x0 e XS and an LTL task φ, learn a safe optimal policy π* such that:

[0024] a. under the policy π*, M is an initial state opaque sequence generated from x0;

[0025] b. the following equation is maximized:

[0026] i.e. the agent walks out of the trajectory p from the state x0 with the policy π*, the sequence seen by the observer is which satisfies the LTL task φ, while the total reward value obtained by the agent is maximized; Pr represents the optimization task;

[0027] Given a MDP M = (X, A, X0, T, AP, L), given a secret state set and an output function H: X -> Y, the estimate of the initial state after observing a trajectory H(p) is defined as:

[0028]

[0029] where p is the path actually taken by the agent, x[i] represents the state of the agent at each time, Path M is the set of paths that the MDP M can generate, and p' is the path such that the intruder can observe the same output sequence as PathM produces, H(p) is the output sequence of the path p observed by the observer, is the set of paths consisting of such paths.

[0030] As a preferred, in step 5), define the initial state estimator: given a MDP M = (X, A, X0, T, AP, L), a secret state and an output function H: X -> Y, the initial state estimator is defined as:

[0031] ISE(M) = (M, Y, δise, M0), where M is a finite set of states; Y is a finite set of outputs; M0: 2X0 is a set of initial states, for any m0 e M0, if (x10, x10) e m0 and (x20, x20) e m0, then H(x10) = H(x20); δise: M x Y -> M is a transition function defined by the following method: for any m, m' e M, y e Y, where m = (x1, x2), then the following equation holds:

[0032] m' = δ ise (m,y)

[0033] For all states m' e M of the initial state estimator ISE, m' has two components, the first component is a set of initial states X0, and the second component is a set of current states.

[0034] The present application models the relationship between network regions, excludes routing schemes that may leak the initial state to the attacker during routing, thereby ensuring the security of network routing. The present application models the conditions that need to be met during the routing process of the power grid system into the LTL paradigm, trains the network routing algorithm using reinforcement learning, and reasonably integrates security-based considerations into the reinforcement learning process based on the LTL paradigm, thereby enabling effective security perception. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of a security perception method based on a power network for hiding vulnerability attack paths in Example 1;

[0036] Figure 2 Figure illustrating the connectivity between the areas in which messages can be sent in the power network and different areas for example 2;

[0037] Figure 3 Figure illustrating the MDP model for the areas of the power network for example 2;

[0038] Figure 4 Figure illustrating the initial state estimator for the areas of the power network for example 2. DETAILED DESCRIPTION

[0039] For a further understanding of the present application, reference will be made to the following description taken in conjunction with the accompanying drawings. It is understood that the examples are merely illustrative of the present application and are not limiting.

[0040] Example 1 as shown in the following table provides a security-aware approach for vulnerability attack path hiding based on a power network, which comprises the following steps: Figure 1

[0041] 1) Model Markov Decision Processes (MDPs) with security constraints; MDPs generally involve an agent interacting with an environment, and the possible states, actions, and rewards in this environment. In this example, the modeling of MDPs will focus on the security issues when the power network routes messages, especially on protecting the initial areas from being leaked.

[0042] In step 1), an MDP is the environment faced by an agent, which is modeled as: M = (X, A, X0, f, AP, L), where M represents the MDP Markov Decision Process model, X is a finite set of states; A is a finite set of actions; X0is a finite set of initial states, i.e., the initial state of the agent is a subset of all possible states; f: X x A x X -> [0, 1] is a state transition function; L is a labeling function.

[0043] Given an MDP M, a path x0∈ X0is a sequence

[0044]

[0045] ​where x[i] represents the state of the agent at the i-th time, x[1] = x0, i.e., the agent is in the initial state x0 at the first time; the policy π satisfies f(x[i], π(x[1]a[1]...x[i]), x[i+1]) > 0, where π(x[1]a[1]...x[i]) represents a series of policies taken by the agent, i.e., taking action a[1] policy at state x[1], taking action a[2] policy at state x[2], and so on until moving to state x[i]; f(x[i], π(x[1]a[1]...x[i]), x[i+1]) represents a state transition probability, i.e., the probability that the agent moves to x[i+1] after taking policy π(x[1]a[1]...x[i]) at state x[i], X ω is the set of all infinite sequences over X. 2) Model the linear temporal logic LTL specification; LTL is a logic language for describing the temporal changes of system behavior, which allows us to express the properties that the system should satisfy. In the security-aware approach, the LTL specification will be used to define the security properties that the power network routing should comply with, such as preventing the leakage of initial region information.

[0046] In step 2), the LTL formula φ over the set of atomic propositions AP is formed according to the following syntax:

[0047]

[0048] where φ represents a logical task model, a is an atomic proposition, φ1 and φ2 represent sub-logical task models, represents the next step, U represents until, and true represents true result;

[0049] For an MDP M, a trace is a sequence:

[0050] t = H(p) = H(x[0])H(x[1])H(x[2])... ∈ H(x) ω such that f(x[i], π(x[1]a[1]...x[i]), x[i+1]) > 0 for all i, where H is a marking function, representing the state of the agent seen by the observer (the observer cannot see the exact state of the agent, but can only confirm that the agent is currently in one or more states), for example, H(x[1]) represents the state of the agent seen by the observer at state x[1] (the state at time 1), H(p) represents the trace of the agent seen by the observer from state x[0] to the end, and ω represents an infinite sequence.

[0051] 3) Model the invader attack model, the opacity of the initial state; opacity is a security property that describes the degree of invisibility or difficulty of inferring certain information of the system (in this embodiment, the initial region) to an external observer (such as a potential invader). By modeling the opacity of the initial state, the risk of the power network routing leaking initial region information when sending messages can be quantitatively evaluated. The modeling of the invader attack model helps us design more effective security strategies to counter potential attacks.

[0052] In step 3), the initial state opacity is modeled to model security requirements, given an MDP M = (X, A, X0, f, AP, L), assume there is a set of secret states X0 represents all possible states, Xs represents all possible initial states, and the invader can observe the output sequence of the system, which is generated by the output function H: X→ Y, where Y is the output set, and H can be extended to H(p) = H(x[1]) H(x[2])…;

[0053] Under the policy π, the initial opaque state about x0∈XS is defined as:

[0054] H(s) = H(t), that is, for a path 1 consisting of a series of state sequences starting from the state x[1] = x0 (the state x0 at the first time), if there is a path 2 different from path 1 starting from a state in the set X0 (the set of initial states), that is, each state x’[i] in path 2 can be different from the state x[i] in path 1, in theory, the agent has a certain probability to walk path 2, and the observer sees the same state at each time of path 1 and path 2 (the observer sees the state is not the exact state, but one or more possible states), then the initial state x0 is opaque.

[0055] 4) Model the reinforcement learning problem under security constraints;

[0056] In step 4), the reinforcement learning problem under security constraints is modeled:

[0057] Given an MDP M and a secret state An initial state x0∈XS and an LTL task φ, learn a safe optimal policy π* such that:

[0058] a. Under the policy π*, M is the initial state opaque sequence generated from x0; b. The following formula is maximized:

[0059] That is, the agent takes the policy π* from the state x0 and follows the trajectory p, and the observer sees the sequence It satisfies the LTL task φ while the total reward value the agent gets is maximized; Pr denotes the optimization task.

[0060] Given an MDP M = (X, A, X0, T, AP, L), given a secret state set and an output function H: X → Y, the estimate of the initial state after observing the string H(p) ∈ Y* is defined as:

[0061]

[0062] where p is the path the agent actually takes, x[i] represents the state of the agent at each time, Path M is the set of paths that the MDP M can generate, and p' is the path that makes it possible for the intruder to observe the same output sequence as PathM generates (i.e., the sequence observed by the observer is the same as the output sequence generated by PathM), H(p) is the output sequence of the path p observed by the observer, is the set of paths consisting of such paths.5) Construct the initial state estimator ISE of the MDP, build the task model; the final strategy should satisfy the security specification of the initial state opacity. One feasible method to meet this requirement is to find all possible paths that can leak the initial state to the intruder in this system. After that, all operations that can lead the system to enter these unsafe paths can be disabled. To this end, one of the key components of the present embodiment is to track the information flow of the intruder, and a complete method is to construct the entire initial state estimator (ISE) of the MDP.

[0063] Therefore, in step 5), the initial state estimator is defined: given an MDP M = (X, A, X0, T, AP, L), a secret state and an output function H: X → Y, the initial state estimator is defined as:

[0064] ISE(M) = (M, Y, δise, M0), where M is a finite set of states; Y is a finite set of outputs; M0: 2X0⊙X0 is a set of initial states, for any m0∈M0, if (x10, x10) ∈ m0 and (x20, x20) ∈ m0, then H(x10) = H(x20); δise: M × Y → M is a transition function defined by the following method: for any m, m' ∈ M, y ∈ Y, where m = (x1, x2), the following equation holds:

[0065] m' = δ ise (m,y)

[0066] For all states m' ∈ M of the initial state estimator ISE, m' has two components (as m and y in the above equations, respectively), the first component set is the initial state X0, and the second component set is the current state.

[0067] 6) Solve with Q-learning to get the optimal policy, i.e., complete the security perception. Reinforcement learning is a method of learning how to make decisions through trial and error, which is particularly suitable for solving decision-making problems with complex environments and uncertainties. In the security perception method, reinforcement learning will be used to train a policy that can optimize the routing of messages in the power network under the condition of meeting the LTL specification and security constraints. This policy will consider how to balance the transmission efficiency of messages and the security of the initial region to ensure that the information of the initial region can be effectively protected even in the face of potential attacker attacks.

[0068] This embodiment models the relationship between network regions and excludes routing schemes that may leak the initial state to the attacker during routing, thereby ensuring the security of network routing. This embodiment models the conditions that need to be met during the routing process of the power grid system as an LTL paradigm, and uses reinforcement learning to train the network routing algorithm. Security-based considerations are reasonably integrated into the reinforcement learning process based on the LTL paradigm, thereby enabling effective security perception.

[0069] Embodiment 2 This embodiment considers a power network coverage area divided into different regions, as shown in Figure 2 All regions are divided into two network regions g and w according to different network node characteristics, where C, F belong to the g region, and A, B, D, E belong to the w region. The network node sends information from region (A) and needs to send information to regions D and F infinitely frequently. There are message transmission links between each region, which are divided into one-way links (feasible directions are shown by arrows) and two-way links (channels without arrows). Each region can send information to regions in four directions, denoted as "r (right)", "l (left)", "u (up)", and "d (down)". It is assumed that there is a small error when the network node sends messages. For example, when the network node in region A sends a message to the right region, it may enter one of the two regions: C (g region) or B (w region).

[0070] Suppose there is an external intruder who wants to determine from which region (A or E) the network node sends the message. Each region has different network characteristics, and when a message is sent to that region, the intruder can observe the relevant network characteristics. The network regions can be divided into w-regions and g-regions according to different network characteristics, and the intruder can observe that the message entered a w-region or a g-region, but does not know which region specifically. When a message is sent to a new region, the intruder can observe that the message entered a w-region or a g-region, and can infer from this the possible initial region from which the message was sent in the power network. The fullness constraint requires that the final strategy should preserve the opacity of the initial state.

[0071] To solve this problem, the workspace can be converted into an MDP as shown in Figure 3 The network sends a message from region A. The user wants it to send messages to regions D and F infinitely often (there are nodes in D and F that need to receive network messages frequently), and only visit region C a finite number of times (region C belongs to a transition region, and the user does not want the network node to send information to region C frequently because it needs to consume network resources or can leak information, etc.). For example, in region A, the network node can choose action r or 1 (send the message to the right or left direction), and if it chooses r, then the message can be sent to B or C, and the transition probability of sending the message is unknown. The intruder knows that the message can be sent from A or E, and can infer from the regions that the message has passed through to guess where the message was originally sent from, and the user does not want the intruder to know the specific region from which the message was sent. Now, suppose that the message was originally released from A. One possible path in this system is where the superscript ω means a finite number of repetitions. This path is insecure because the intruder can infer from the observed output (w→w→w→g)ω that the message was originally sent from A, because if the message was originally sent from E, there is no feasible way to generate the same observation sequence. One possible secure path is Because there is another possible path that sends the message from region E originally has the same output sequence (w→g→w→w)ω, i.e., the intruder cannot infer from the observation sequence (w→g→w→w)ω that the message was originally sent from A or E. By analogy, one possible path that not only satisfies the LTL specification but also keeps the initial state opaque is where the superscript * means a finite number of repetitions.

[0072] The final policy should satisfy the security specification with initial state opacity. One possible way to satisfy this requirement is to find all possible paths in the system that leak the initial state to the intruder. After that, we can disable all operations that can lead the system to these unsafe paths. To do this, one of the key components of the algorithm in this paper is to track the intruder's information flow, and one complete way to do this is to construct the entire initial state estimator (ISE) of the MDP.

[0073] Given an MDP M = (X, A, X0, T, AP, L), given a set of secret states and an output function H: X → Y, the initial state estimate after observing a string H(p) ∈ Y* is defined as:

[0074]

[0075] where PathM is the set of paths that M can generate, and is the set of all paths p' such that the intruder can observe the same output sequence as PathM, in the above example, the same output sequence is specifically the sequence that allows the intruder to observe the same w or g signal. The path p' can be any path that the MDP can generate without considering a specific policy, because the observer does not know the specific area that the message has passed through.

[0076] However, it is impossible to store all observation sequences directly, because the number of observation sequences can be very large, so we must map the observation sequences onto a finite structure. For a finite state set X, define the operator 2 : X → Xto represent the state X ⊙ X := {(x, x) | x ∈ X}, where two states represent the starting area of the path and the final area of the path, respectively. To define the state transition in the initial state estimator, define the composition operator for where m is the state mapping relationship. Finally, define the mapping as

[0077]

[0078] O(α) represents the set of paths that produce and observe the same head and tail as the sequence . Define the initial state estimator: Given an MDP M = (X, A, X0, T, AP, L), a set of secret states and an output function H: X → Y, the initial state estimator is defined as:

[0079] ISE(M) = (M, Y, δ iseM0)

[0080] where M is a finite set of states; Y is a finite set of outputs; M0:2 X0⊙X0 is the set of initial states, for any m0∈M0, if (x 10 , x 10 )∈m0 and (x 20 , x 20 )∈m0, then H(x 10 ) = H(x 20 ); δ ise : M x Y→M is a transition function defined by the following method: for any m, m’∈M, y∈Y, where m = (x1, x2), then the following equation holds:

[0081] m' = δ ise (m, y)

[0082] For all states m’∈M of the initial state estimator ISE, m’ has two components (as m and y in the above equations, respectively), the first component is the set of initial states X0, and the second component is the current state. For Figure 3 the MDP shown in Figure 4 the initial state estimator of this MDP is shown. Assume the initial uncertainty is equal to the initial state space, so m0= δ ise (ε, w) = {(A, A), (E, E)}. After observing g, the state becomes i.e., after the intruder observes the g signal, the possible path is either (A, C) or (E, C), so the initial state cannot be confirmed to be A or E, i.e., the initial state is not leaked. Similarly, on the other branch, i.e., after observing the w signal, the state becomes i.e., after the intruder observes the w signal, the possible path at this time is (A, B), because there is no path from E to the second step that emits the w signal, i.e., the intruder can confirm that the initial state is A. Continuing this process, the ISE of the entire system can be constructed completely in Figure 2 as shown in Figure 4 .

[0083] The task of this embodiment is to implement a controller of the MDP that jointly synthesizes the security constraints and the LTL task. The above steps will be summarized into the following algorithm:

[0084] The following is a specific description of the algorithm:

[0085] First, we construct the task model from the original MDP, the initial state estimator. Then, we solve the problem using reinforcement learning (Q-learning).

[0086] 1) Construct the task model from the original MDP, initial state estimator: Note that during training, an episode does not always end when the agent first reaches an acceptable state, because LTL requires the message to repeatedly reach the acceptable state. Therefore, the agent's exploration in this round will only end when it reaches state t (i.e. the message reaches certain specific regions enough times).

[0087] According to the LTL specification, let the atomic propositions AP = {AreaD, AreaF, AreaC}, define L(D) = AreaD, L(F) = AreaF, L(C) = AreaC, for other states besides D, C, F, Then the task to be completed by the power network routing can be expressed as

[0088]

[0089] That is, after a limited number of rounds of training, the message intermittently and continuously reaches the D and F regions, but no longer reaches the C region. In order to ensure the opacity of the initial state, the message must first reach the C region. After that, it can move arbitrarily, frequently reaching the D and F regions through wireless, obtaining infinite rewards, and in the final strategy, it will access the D and F regions infinitely frequently.

[0090] 2) Solve the problem using reinforcement learning (Q-learning):

[0091] In the algorithm, there is That is, the Q value is a binary tuple about state and action A safe . It measures the potential future rewards the agent will obtain when choosing action a in state x. During training, is chosen by in state to choose action This means that the action is selected from the q values of all possible actions in state (some safe state) (see line 14). The q value will be updated in a round (see line 16), and γ ∈ [0, 1] is a discount factor, the larger γ, the more attention is paid to the "long-term earnings" the agent can obtain. In practice, it is not possible to simulate the case where the network node obtains infinite rewards. A very large reward is set at state t (the agent reaches certain specific regions in the environment many times), and this round is updated. Among them, Acc represents the state in which the agent can obtain a large reward. Therefore, the message in each region tends to be sent in a certain direction, i.e. θ (in reinforcement learning, this letter represents the probability that the agent will take this action) approaches 1 when the task is completed. When the maximum number of rounds is completed, the optimal strategy for state is to maximizing the action a, thereby maximizing An optimal policy π* that satisfies a certain LTL requirement φ under the MDP model M.

[0092] By the above method, the embodiment designs a safety-aware reinforcement learning solution under linear temporal logic specification (LTL), gives an explicit modeling for the LTL problem involving safety and privacy in reinforcement learning, and proposes an effective solution.

[0093] The above describes the present application and its embodiments in a schematic manner, which is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired thereby, without departing from the purpose of the present application, similar structural modes and embodiments are not creatively designed, which should belong to the protection scope of the present application.

Claims

1. A security-aware method based on power network usable for vulnerability attack path concealment, characterized in that: The steps include: 1) modeling Markov Decision Processes (MDPs) with security constraints; 2) modeling Linear Temporal Logic (LTL) specifications; 3) modeling an intruder attack model, opacity of initial states; 4) modeling reinforcement learning problems under security constraints; 5) constructing an initial state estimator (ISE) for MDPs, building a task model; 6) solving with Q-learning to obtain an optimal policy, i.e., completing security perception; In step 1), an MDP is an environment faced by an agent, which is modeled as: M = (X, A, X0, f, AP, L), where M represents the MDP Markov Decision Process model, X is a finite set of states; A is a finite set of actions; X0 represents all possible states; is a finite set of initial states, i.e., the initial state of the agent is a subset of all possible states; f: X x A x X→ [0,1] is a state transition function; L is a labeling function; Given an MDP M, a path x0∈X0 is a sequence Wherein, x[i] represents the state of the agent at i moment, x[1]=x0, that is, the agent is in the initial state x0 at the first moment; The policy π satisfies f(x[i], π(x[1]a[1]...x[i]), x[i+1])>0, wherein π(x[1]a[1]...x[i]) represents a series of strategies adopted by the agent, that is, the action a[1] strategy is taken at state x[1], the action a[2] strategy is taken at state x[2], and so on, until the state x[i] is transferred; f(x[i], π(x[1]a[1]...x[i]), x[i+1]) represents a state transition probability, that is, the probability that the agent is transferred to x[i+1] after taking the policy π(x[1]a[1]...x[i]) at state x[i], X ω is the set of all infinite sequences on X; In step 3), the security requirements are modeled in terms of initial state opacity, given a MDP M = (X, A, X0, f, AP, L), assume a set of secret states Xs represents all possible initial states, an intruder can observe the output sequence of the system, which is generated by the output function H: X → Y, where Y is the output set, H is extended to H(p) = H(x[1]) H(x[2])...; Under a policy π, the initial opaque state with respect to x0∈Xs is defined as: such that H(s) = H(t); In step 4), modeling reinforcement learning problems under security constraints: Given an MDP M with secret states an initial state x0∈ Xs and an LTL task φ, learn a secure optimal policy π* such that: a. M is an initial state opaque sequence generated from x0 under a policy π*; b. The following formula is maximized: i.e., the agent takes policy p from state x0and follows the trajectory p, the sequence observed by the observer is It satisfies the LTL task φ while the total reward value received by the agent is maximized; Pr denotes the optimization task; Given an MDP M = (X, A, X0, T, AP, L), given a secret state set and an output function H: X → Y, the estimate of the initial state after observing the trajectory H(p) is defined as: where p is the path actually taken by the agent, x[i] represents the state of the agent at each time, Path M is the path set that MDP M can generate, and p' is the path that makes the intruder can observe the same output sequence as PathM, H(p) is the output sequence of the path p observed by the observer, is the path set composed of such paths.

2. The power grid based security aware approach for vulnerability attack path concealment as claimed in claim 1 wherein: In step 2), an LTL formula φ over an atomic proposition set AP is formed according to the following grammar: Where φ represents a logical task model, a is an atomic proposition, φ1 and φ2 represent sub-logical task models, ○ represents the next step, U represents until, and true represents a true result; For an MDP M, a trace is a sequence: t = H(p) = H(x[0]) H(x[1]) H(x[2])... e H(x) ω such that f(x[i], π(x[1] a[1]... x[i]), x[i+1]) > 0 for all i, where H is a labeling function representing the state of the agent as seen by the observer, H(p) represents the trajectory of the agent from state x[0] to the end as seen by the observer, and ω represents an infinite sequence.

3. The power grid based security aware approach for vulnerability attack path concealment as claimed in claim 2 wherein: In step 5), the initial state estimator is defined: Given an MDP M = (X, a, X0, T, AP, L), a secret state and an output function H: X → Y, the initial state estimator is defined as: ISE(M) = (M, Y, δise, M0), where M is a finite set of states; Y is a finite set of outputs; M0: 2 X0⊙X0 is a set of initial states, for any m0 e M0, if (x10, x10) e m0 and (x20, x20) e m0, then H(x10) = H(x20); δise: M x Y → M is a transition function defined by the following method: for any m, m' e M, y e Y, where m = (x1, x2), then the following equation holds: m' = δ ise (m, y); For all states m'∈M of the initial state estimator ISE, m' has two components, the first component set is the initial state X0, and the second component set is the current state.

Images