A method for vulnerability attack path planning based on reinforcement learning

Abstract

The application relates to the technical field of network security, and provides a method for planning a vulnerability attack path based on reinforcement learning, which comprises the following steps: step 1, establishing a robot autonomous path planning safety reinforcement learning framework SSRL, which comprises two parts: 1.1, a safety guarantee mechanism for predicting and prohibiting unsafe behaviors; and 1.2, a self-recovery mechanism capable of recovering to a previous safe state; step 2, robot path planning based on dynamic support set small sample reasoning type safety reinforcement learning, which comprises two parts: 2.1, a safety exploration part; and 2.2, a self-recovery part; and step 3, fast reinforcement learning robot path planning based on an experience reuse mechanism. The application can preferably plan a vulnerability attack path.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and more specifically, to a method for vulnerability attack path planning based on reinforcement learning. Background Technology

[0002] The dynamic and complex network environment enhances the stealth and harm of network attacks, and the increasingly serious network attack threats pose a severe challenge to existing network intrusion detection systems. Among these challenges, path planning is a key method in network security, and an excellent path planning method can solve security problems such as error recovery and training security in unknown environments.

[0003] However, current path planning methods cannot adequately address the safety issues of learning and training, and struggle to resolve error recovery problems after unavoidable failures occur. Summary of the Invention

[0004] The present invention provides a method for vulnerability attack path planning based on reinforcement learning, which can solve the security problem of learning and training, solve the error recovery problem after unavoidable failures occur, and take advantage of the strong adaptability of reinforcement learning to quickly relearn in dynamic environments that are closer to actual production and life, plan reasonable paths, improve work efficiency, and save costs.

[0005] According to a method for vulnerability attack path planning based on reinforcement learning according to the present invention, the method includes the following steps:

[0006] Step 1: Establish a robot autonomous path planning safety reinforcement learning framework (SSRL), which includes two parts:

[0007] 1.1 A safety assurance mechanism for predicting and prohibiting unsafe behaviors, employing a shield tunneling mechanism-based safety assurance system;

[0008] 1.2 A self-recovery mechanism capable of restoring the user to a previous safe state;

[0009] Step 2: Few-shot reasoning-based safe reinforcement learning robot path planning based on dynamic support sets, consisting of two parts:

[0010] 2.1 In the security exploration section, few-shot learning with a dynamic support set is used to identify previously unseen vulnerabilities and eliminate unsafe actions in future explorations;

[0011] 2.2 Self-recovery component: When a failure occurs, it can revert to the previous state and reflect on the failure to update the support set;

[0012] Step 3: Rapid reinforcement learning robot path planning based on experience reuse mechanism.

[0013] As a preferred option, the specific implementation method of the security mechanism is as follows:

[0014] First, we define a set of critical states to represent all critical locations of the vulnerability, as follows:

[0015]

[0016] Where S obs It is a collection of all vulnerabilities;

[0017] Next, define the set of dangerous actions:

[0018]

[0019] A dangerous action can only be selected from a set of critical states; in this definition, unsafe behavior in a critical state leads to unsafety; when the robot performs a dangerous action, the safety mechanism can calculate other critical positions around the specific vulnerability corresponding to the critical state, and use this to predict more unsafe actions; at time t, the predicted surrounding states of the specific vulnerability and the dangerous actions corresponding to each state are represented as follows:

[0020]

[0021] Among them, S obs It is an unknown vulnerability set S obs A specific data point, S pc and a pd s obs Other critical states in the surrounding area and their corresponding dangerous actions;

[0022] To ensure the exploration complies with safety regulations, a shield mechanism is introduced to monitor the reinforcement learning training process. Initially, the shield does not store any environment-related safety information. During the agent's exploration, if a collision occurs, the shield adds the failed state-action pair to its memory for safety checks of subsequent actions. When the agent moves to a critical state and violates safety regulations (i.e., takes a dangerous action in that state), the shield is activated to prohibit the unsafe action and reselects an action until it is safe. Simultaneously, the shield mechanism calculates other state-action pairs around the vulnerability based on the failed state-action pair and updates them to the safety checks. When the robot moves to these states and performs the corresponding action, the safety shield is activated again. Therefore, the path planning problem can be summarized as follows:

[0023] Maximize the action value function:

[0024] Simultaneously satisfy constraints

[0025] Q π (s,a) is the long-term cumulative expectation that the Q-learning algorithm needs to optimize. The agent's goal is to maximize Q. π (s,a) without violating safety regulations, A pd Indicates vulnerability S obs Other critical states in the surrounding area and their corresponding dangerous actions, A d This represents a set of dangerous actions; the safety mechanism guides action selection by monitoring the agent.

[0026] As a preferred option, the specific implementation method of the self-recovery mechanism is as follows:

[0027] The robot's motion trajectory can be represented as the following discrete sequence of states and actions:

[0028] τ={S1,a1,S2,a2,…,s t ,a t}

[0029] At time t, the state is s. t The robot performs the action a t Then convert to s t+1 This state is a vulnerability that could lead to a collision. Based on environmental feedback, the security mechanism will change the state s upon detecting a failure. t The following action a t In addition, rule-based predicted state-action pairs are added to the tunnel boring machine; the self-recovery mechanism aims to revert the robot to safe states s1, s2, ..., s in the past sequence without changing the reinforcement learning framework. t One of the measures is that when danger occurs, i.e. the robot collides with a vulnerability, the robot first updates the value of the action using the Bellman equation, then takes the opposite action to the action in the previous state to return to the previous state, which is a safe state. Finally, the robot can select actions normally.

[0030] As a preferred option, the specific implementation method of step 2 is as follows:

[0031] The support set is dynamic, based on the critical state set and the dangerous action set (s tc ,a td (s) represents a specific critical state and the path planning towards a specific vulnerability in that state; tc ,a td This refers to important hidden security information in the environment. The responsibility of the security exploration part, in path planning, is to monitor the agent to prevent it from accessing sensitive information. tc Select a in the state td Security information has two path planning sources:

[0032] 1. From the few-sample reasoning module, if the module correctly identifies the vulnerability, it indicates that a collision will not occur during path planning, and the robot is at the critical position s. tc Towards vulnerabilities; the security exploration section will collect location information for the robot's path planning and prevent the robot from taking actions. td To prevent collisions;

[0033] Second, if the small sample reasoning module identifies the path planning error, it means that the robot will collide with the vulnerability. The self-recovery part collects vulnerability information of the real path planning after the collision to ensure safety after the collision.

[0034] As a preferred option, the specific implementation method of step 3 is as follows:

[0035] 3.1 Under the strategy before the dynamic changes in the environment path planning, determine the distribution of vulnerabilities on the original path, grasp the pattern of vulnerability changes, and thus determine the scope of path planning to be updated.

[0036] 3.2. Within the scope of the re-update, determine the starting point and the ending point, explore the area, and derive the change strategy after the dynamic changes in the environment;

[0037] 3.3 Integrate the original strategy with the new strategy to obtain the final strategy and verify the path planning.

[0038] This invention addresses the error recovery problem in reinforcement learning path planning applications by proposing a safe, self-recovering autonomous path planning framework for reinforcement learning. This framework expands and predicts unsafe state-action pairs based on actual path planning algorithms to ensure robot safety during exploration. Furthermore, after an error occurs, a recoverable state space allows the robot to revert to a previous safe state, reducing the number of times the robot needs to be reset due to errors, thus facilitating practical deployment of reinforcement learning.

[0039] Furthermore, addressing the learning security issues in practical path planning applications of reinforcement learning, a path planning method based on few-shot reasoning-based secure reinforcement learning is proposed. This method, building upon the aforementioned security framework, combines few-shot learning with security assurance mechanisms for reasoning about unknown vulnerabilities and related dangerous actions. Additionally, after an error occurs, dynamic management of the support set is incorporated into the self-recovery mechanism to enhance the reasoning capability for such vulnerabilities and reduce the occurrence of errors.

[0040] Furthermore, addressing the relearning efficiency issue in practical path planning applications of reinforcement learning, a fast reinforcement learning robot path planning method based on an experience reuse mechanism is proposed. By exploring within a smaller region of re-updates, the policy search scale is reduced, thereby achieving the goal of rapid learning.

[0041] This invention specifically studies a robot attack path planning method based on reinforcement learning, focusing on path planning in unknown dynamic environments. Addressing the security and self-recovery issues during reinforcement learning deployment in unknown environments, it proposes a secure and self-recoverable reinforcement learning autonomous path planning framework. Furthermore, it proposes a path planning method based on few-shot reasoning-based secure reinforcement learning, supplemented by a fast reinforcement learning robot path planning method based on experience reuse mechanisms. This approach ensures the safety of reinforcement learning-based robots exploring unknown environments while accelerating the relearning rate after environmental changes. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for vulnerability attack path planning based on reinforcement learning in an embodiment.

[0043] Figure 2 This is a schematic diagram of the secure self-recovery reinforcement learning path planning framework in the embodiment.

[0044] Figure 3 This is a schematic diagram of the safety guarantee mechanism design based on tunnel boring machines in the embodiment.

[0045] Figure 4 This is a schematic diagram of the self-recovery mechanism in the embodiment.

[0046] Figure 5 This is a schematic diagram of security reinforcement learning based on few-sample reasoning in the embodiment.

[0047] Figure 6 This is a flowchart of the small-sample reasoning-based secure reinforcement learning path planning method in the embodiment.

[0048] Figure 7 This is a flowchart illustrating the steps of the experience reuse mechanism in the embodiment. Detailed Implementation

[0049] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0050] Example

[0051] like Figure 1As shown, this embodiment provides a method for vulnerability attack path planning based on reinforcement learning, which has three core steps: First, expand and predict state-action pairs based on robot path planning experience to ensure the robot's exploration safety, and after an error occurs, use the recoverable state space to enable the robot to fall back to the previous safe state for practical deployment of reinforcement learning; Second, based on the above security framework, combine the few-shot learning method with the security assurance mechanism for reasoning about unknown vulnerabilities and related dangerous actions; Third, to address the relearning efficiency problem in practical reinforcement learning path planning applications, a fast reinforcement learning robot path planning method based on an experience reuse mechanism is proposed to reuse the experience pool outside the scope.

[0052] Specifically:

[0053] Step 1, as follows Figure 2 As shown, a safe and self-recoverable reinforcement learning framework (SSRL) for robot autonomous path planning is established, which extends the standard reinforcement learning framework by two parts:

[0054] 1.1 A safety assurance mechanism for predicting and prohibiting unsafe behaviors, employing a shield tunneling mechanism-based safety assurance system;

[0055] The specific implementation methods of the security guarantee mechanism are as follows:

[0056] To ensure robot safety, a set of critical states is first defined to represent all critical locations of the vulnerability, as follows:

[0057]

[0058] Where S obs It is a collection of all vulnerabilities;

[0059] Next, define the set of dangerous actions:

[0060]

[0061] A dangerous action can only be selected from a set of critical states; in this definition, unsafe behavior in a critical state leads to unsafety; when the robot performs a dangerous action, the safety mechanism can calculate other critical positions around the specific vulnerability corresponding to the critical state, and use this to predict more unsafe actions; at time t, the predicted surrounding states of the specific vulnerability and the dangerous actions corresponding to each state are represented as follows:

[0062]

[0063] Among them, sobs It is an unknown vulnerability set S obs A specific piece of data, s pc and a pd s obs Other critical states in the surrounding area and their corresponding dangerous actions;

[0064] To ensure that the exploration meets safety standards, a tunneling mechanism was introduced to monitor the reinforcement learning training process, such as... Figure 3 As shown; initially, the tunnel boring machine (TBM) does not store any safety information related to the environment. During the agent's exploration, if a robot encounters a conflict, the TBM adds the failed state-action pair to the TBM for safety checks of subsequent actions. When the agent moves to a critical state, if it violates safety rules (i.e., takes a dangerous action in this state), the TBM is activated to prohibit this unsafe action and reselects an action until it is safe. Simultaneously, the TBM mechanism calculates other state-action pairs around the vulnerability based on the state-action pair that caused the failure, and updates them to the safety checks. The next time the robot moves to these states and performs the corresponding action, the safety TBM is activated. Therefore, the path planning problem can be summarized as:

[0065] Maximize the action value function:

[0066] Simultaneously satisfy constraints

[0067] Q π (s,a) is the long-term cumulative expectation that the Q-learning algorithm needs to optimize. The agent's goal is to maximize Q. π (s,a) without violating safety regulations, A pd Indicates vulnerability S obs Other critical states in the surrounding area and their corresponding dangerous actions, A d This represents a set of dangerous actions; the safety mechanism guides action selection by monitoring the agent.

[0068] More specifically, if the agent samples a dangerous action from the safety assurance module during the action selection process, the safety assurance module will prohibit the unsafe action and reselect a safe action. The action selection strategy is similar to Q-learning, but it is optimized for the confidence action space in the critical state.

[0069] 1.2 A self-recovery mechanism capable of restoring the user to a previous safe state;

[0070] When a vulnerability detection robot encounters a vulnerability during training, a natural way for reinforcement learning methods to handle this emergency is to end the current round and start a new one. However, in practical deployments of reinforcement learning on autonomous robots, it's not always possible to restart training from the initial position. Therefore, this embodiment develops a self-recovery mechanism to return to the previous safe state instead of resetting the robot's position to the initial one. Figure 4 As shown, the specific implementation method of the self-recovery mechanism is as follows:

[0071] The robot's motion trajectory can be represented as the following discrete sequence of states and actions:

[0072] τ={s1,a1,s2,a2,…,s t ,a t}

[0073] At time t, the state is s. t The robot performs the action a t Then convert to s t+1 This state is a vulnerability that could lead to a collision. Based on environmental feedback, the security mechanism will change the state s upon detecting a failure. t The following action a t In addition, rule-based predicted state-action pairs are added to the tunnel boring machine; furthermore, in actual deployment, if the robot collides with a vulnerability, a natural idea is to back off the robot and re-determine the situation; the self-recovery mechanism aims to revert the robot to safe states s1, s2, ..., s in the past sequence without changing the reinforcement learning framework. t One of the issues arising from this is which state to revert to for the best overall exploration. Figure 4 Assuming that returning to state s2 is the optimal state, which is more conducive to subsequent exploration, the robot can recover to state s2 according to the self-recovery strategy and reselect action a2' to execute in the environment. In this embodiment, the self-recovery mechanism only considers recovering to the most recent state, i.e., s2. t This is because it consumes less memory. Therefore, when a danger occurs—that is, when the robot collides with a vulnerability—the robot first updates the value of the action using the Bellman equation, then takes the opposite action to the action in the previous state to revert to that state, which is a safe state. Finally, the robot can select an action normally. However, it's important to note that due to the safety mechanism, after a sample of the action is collected after the collision, it will not be sampled again after reverting to the previous state. By restoring to the previous safe state, the robot's exploration is closer to reality, and theoretically, there will be fewer failures.

[0074] Step 2: Few-shot reasoning-based safe reinforcement learning robot path planning based on dynamic support sets.

[0075] The self-recovering reinforcement learning autonomous path planning framework proposed in Step 1 provides a safety guarantee for the practical deployment of reinforcement learning on autonomous robots. However, the prediction part in the tunneling mechanism is rule-based and has high limitations. In unknown environments, it is not always possible to assume that the predicted information can be obtained. Therefore, Step 2, from the perspective of vulnerability identification, attributes the above-mentioned predicted information to the robot's reasoning on the captured vulnerability image. At the same time, the self-recovery mechanism in the framework allows the robot to continue exploring after failure. However, in fact, failures often contain more information about security risks, which may be helpful for future safe exploration. Therefore, robots that make decisions based on reinforcement learning methods should ideally have the ability to adapt to diverse situations and reflect on failures to further explore safely.

[0076] like Figure 5 and Figure 6 As shown, step 2 consists of two parts:

[0077] 2.1 In the security exploration section, few-shot learning with a dynamic support set is used to identify previously unseen vulnerabilities and eliminate unsafe actions in future explorations;

[0078] 2.2 Self-recovery component: When a failure occurs, it can revert to the previous state and reflect on the failure to update the support set;

[0079] The specific implementation method of step 2 is as follows:

[0080] In practical path planning, any image of a vulnerability captured by the robot from any direction belongs to the path planning query set and is used as a test image for identification by a few-shot inference model. The relationship network combines feature maps of samples in the support set and those in the query set to calculate relationship scores between different categories in the support set, and then selects the category with the highest score to complete the classification problem. In this embodiment, the support set is dynamic, based on the critical state set and the dangerous action set, using (s... tc ,a td (s) represents a specific critical state and the path planning towards a specific vulnerability in that state; tc ,a td This refers to important hidden security information in the environment. The responsibility of the security exploration part, in path planning, is to monitor the agent to prevent it from accessing sensitive information. tc Select a in the state td Security information has two path planning sources:

[0081] 1. From the few-sample reasoning module, if the module correctly identifies the vulnerability, it indicates that a collision will not occur during path planning, and the robot is at the critical position s. tcTowards vulnerabilities; the security exploration section will collect location information for the robot's path planning and prevent the robot from taking actions. td To prevent collisions;

[0082] Second, if the small sample reasoning module identifies the path planning error, it means that the robot will collide with the vulnerability. The self-recovery part collects vulnerability information of the real path planning after the collision to ensure safety after the collision.

[0083] All safety information is acquired during training, while path planning is not known prior. The safety assurance module continuously adds (s) tc ,a td This gradually expands the robot's understanding of safety regulations in the environment. The safety exploration part here is the safety assurance of step 1. The difference is that the prediction information based on the original path planning rules is replaced with the prediction information obtained by the small sample module inference. Only when the path planning vulnerability is correctly identified can this prediction information be conveyed to the safety exploration part and the selection of dangerous path planning actions by the robot be prohibited.

[0084] Step 3: Rapid reinforcement learning robot path planning based on experience reuse mechanism.

[0085] When the environment changes dynamically, the original strategy typically plays a role in path planning for the new strategy to some extent, depending on the magnitude of the change. However, one reason for the slow training speed in reinforcement learning methods is their weak inductive partial path planning ability in response to the environment and their lack of ability to utilize experience pools. Therefore, this embodiment employs a special experience reuse mechanism to overcome this problem.

[0086] like Figure 7 As shown, the specific implementation method of step 3 is as follows:

[0087] 3.1 Under the strategy before the dynamic changes in the environment path planning, determine the distribution of vulnerabilities on the original path, grasp the pattern of vulnerability changes, and thus determine the scope of path planning to be updated.

[0088] 3.2. Within the scope of the re-update, determine the starting point and the ending point, explore the area, and derive the change strategy after the dynamic changes in the environment;

[0089] 3.3 Integrate the original strategy with the new strategy to obtain the final strategy and verify the path planning.

[0090] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for vulnerability attack path planning based on reinforcement learning, characterized in that: Comprising the following steps: Step 1, establish a robot autonomous path planning safety reinforcement learning framework SSRL, including two parts: 1.1, safety guarantee mechanism to predict and prohibit unsafe behavior, using shield mechanism based safety guarantee; 1.2, self-recovery mechanism that can recover to the previous safe state; The specific implementation method of the self-recovery mechanism is: The motion trajectory of the robot is represented as a discrete state and action sequence as follows: t = {s1, a1, s2, a2,..., s t , a t} At time t, the robot performs action a in state s t t and transitions to s t+1 , which is a hole state that will lead to a collision. According to the feedback of the environment, the safety mechanism will shield the action a t in state s t and the state-action pair predicted based on the rules after detecting the failure; the purpose of the self-recovery mechanism is to return the robot to one of the safe states s1, s2,..., s t in the past sequence without changing the reinforcement learning framework; when the danger occurs, i.e., the robot collides with the hole, the robot first updates the value of the action using the Bellman equation, then takes an action opposite to the action in the previous state to return to the previous state, which is a safe state, and finally the robot can normally select an action;​ Step 2: Small sample reasoning based on dynamic support set safety reinforcement learning robot path planning, including two parts: 2.1, safety exploration part, using small sample learning with dynamic support set to identify previously unseen vulnerabilities and exclude unsafe actions in future exploration; 2.2, self-recovery part, when failure occurs, self-back to the previous state, and reflect on the failure to update the support set; Step 3: Fast reinforcement learning robot path planning based on experience reuse mechanism.

2. The method of claim 1, wherein: The specific implementation method of the safety guarantee mechanism is: First, define a critical state set to represent all critical positions of the vulnerability as follows: where S obs is the set of all vulnerabilities; Then, define the dangerous action set: Dangerous actions can only be selected at a certain state in the critical state set; In this definition, unsafe behavior at the critical state leads to unsafe; When the robot performs a dangerous action, the safety guarantee mechanism can calculate other critical positions around the specific vulnerability corresponding to the critical state, which will be used to predict more unsafe actions; At time t, the predicted surrounding states around the specific vulnerability and the dangerous actions corresponding to each state are represented as: where s obs is one data in the unknown vulnerability set S obs , s pc and a pd represent other critical states around s obs and the corresponding dangerous actions; In order to ensure that the exploration meets the safety specification, the shield mechanism is introduced to monitor the reinforcement learning training process; In the initial stage, the shield does not store any safety information related to the environment, during the exploration of the agent, once the robot collides, the shield will add the state-action pair of the failure sample to the shield for safety check of subsequent actions; When the agent moves to a certain critical state, if it violates the safety specification, i.e. takes a dangerous action in this state, the shield will be activated to prohibit this unsafe action and reselect the action until the action is safe; At the same time, the shield mechanism will calculate other state-action pairs around the vulnerability based on the rules according to the state-action pair that leads to failure, and update them to the safety check; When the robot moves to these states next time and performs the corresponding actions, the safety shield will also be activated; Therefore, the path planning problem can be summarized as: Maximizing the action value function: Satisfy constraints simultaneously where Q π (s,a) is the long-term cumulative expectation that the Q-learning algorithm needs to optimize, and the agent's goal is to maximize Q π (s,a) while not violating safety specifications, A pd denotes the set of vulnerabilities S obs surrounding critical states and the corresponding dangerous actions, A d denotes the set of dangerous actions; the safety mechanism guides the action selection by monitoring the agent.

3. The method of claim 2, wherein: The specific implementation method of step 2 is: The support set is dynamic, based on the critical state set and the dangerous action set usage (s tc , a td ) pair to represent a particular critical state and the action planning towards a certain vulnerability in that state; (s tc , a td ) pair is the hidden security information important in the environment, the security exploration part's responsibility is to make sure the agent does not choose a tc in state s td ; the security information has two sources of path planning: I. From the small sample inference module, if the module correctly identifies the vulnerability, it indicates that a collision will not occur in the path planning, and the robot is at a critical position s tc towards the vulnerability; the safety exploration part will collect the position information of the robot path planning and prevent the robot from taking action a td to prevent collision; Two, if the small sample reasoning module identifies the path planning error, it means that the robot will collide with the vulnerability, and the self-recovery part collects the vulnerability information of the real path planning after the collision to ensure safety after the collision.

4. The method of claim 3, wherein: The specific implementation method of step 3 is: 3.1, determine the vulnerability distribution on the original path under the strategy before the dynamic change of the environment, grasp the change rule of the vulnerability, and determine the update range of the path planning; 3.2, determine the starting point and ending point in the update range, and obtain the change strategy after the dynamic change of the environment after exploring the area; 3.3, merge the original strategy with the new strategy to get the final strategy and verify the path planning.

Images