Robustness Enhancement Methods for Deep Reinforcement Learning Navigation Applications Based on Adversarial Maps

Abstract

This invention relates to a robustness enhancement method for deep reinforcement learning (DRL) navigation applications based on adversarial maps, belonging to the field of artificial intelligence security technology. Targeting deep reinforcement learning navigation applications, this invention aims to improve the robustness of DRL navigation systems in real-world environments and ensure safe navigation on various maps. Based on the fundamental principles of adversarial examples, it proposes the definition and generation method of adversarial maps. Building upon this, it designs a robustness enhancement method based on adversarial maps from the perspective of adversarial example attack and defense through an "attack-to-defense" approach. This method addresses the problem of adding adversarial perturbations to the input of intelligent models in real-world environments, providing a usable solution for enhancing the robustness of DRL navigation models in real-world conditions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a robustness enhancement method for deep reinforcement learning navigation applications based on adversarial maps. Background Technology

[0002] In recent years, with the continuous maturation of machine learning theory and technology, the widespread application of artificial intelligence has profoundly impacted people's daily lives. As one of the research hotspots in artificial intelligence technology, Deep Reinforcement Learning (DRL) combines the data feature extraction capabilities of deep learning with the self-learning capabilities of reinforcement learning, and has been extensively studied in fields such as industrial control, robot navigation, game AI, and biomedicine. In mobile robot navigation applications, compared with traditional navigation frameworks that are mostly based on SLAM technology, DRL-based navigation has advantages such as map-free operation, strong learning ability, and low dependence on sensor accuracy. Its end-to-end decision-making method performs well in dynamic or location-based environments, and has received widespread attention from academia and industry.

[0003] On the other hand, while artificial intelligence (AI) technology is developing rapidly, its inherent security issues are gradually attracting researchers' attention, with adversarial examples being a current research hotspot. By adding subtle perturbations to normal samples, adversarial examples can cause the target model to make incorrect decisions. Due to the high complexity and low interpretability of AI systems, it is currently impossible to effectively detect and prevent such security risks. Especially in security-critical fields such as aerospace, industrial control, and autonomous navigation, there are higher requirements for the security and reliability of AI algorithms and models. Therefore, it is necessary to research methods for enhancing the security robustness of intelligent algorithms in these fields.

[0004] Current research on the security of artificial intelligence focuses on supervised learning fields such as image classification and object detection, while work on DRL is still in its early stages. Furthermore, most existing research uses video games as verification scenarios, and the work is only carried out at the algorithm level, meaning the results cannot be directly applied to other fields. Research on the safety of DRL navigation in references 1 (Bai X, Niu W, Liu J, et al. Adversarial Examples Construction Towards White-Box Q Table Variation in DQN Pathfinding Training, in Proceedings of the 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC). Piscataway, NJ:IEEE, 2018:781-787) and 2 (Tong C, Niu W, Xiang Y, et al. Gradient Band-based Adversarial Training for Generalized Attack Immunity of A3C PathFinding. arXiv preprint arXiv:1807.06752, 2018) focuses on pathfinding tasks on grid maps, which differs significantly from real-world navigation environments. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The technical problem to be solved by this invention is: how to add adversarial disturbances to the input of intelligent models in real-world environments, and to provide a usable solution for enhancing the robustness of DRL navigation models in real-world environments.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, this invention provides a robustness enhancement method for deep reinforcement learning navigation applications based on adversarial maps, comprising the following steps:

[0009] Step 1: Based on the fundamental principles of adversarial examples, define adversarial maps and propose adversarial map generation methods;

[0010] Step 2, Security Hardening Training: Using the adversarial map generation method from Step 1, the navigation agent is retrained by generating an adversarial map.

[0011] Preferably, in step one, the DRL navigation system is first defined to include a DRL navigation model, sensors, and a mobile robot. The DRL navigation model refers to the DRL model used for navigation. The sensors are mounted on the mobile robot. The DRL navigation model can make decisions based on the sensor data, enabling the mobile robot to perform various actions. The mobile robot is a navigation agent, referring to a mobile robot that uses the DRL algorithm for navigation. Depending on the type of sensor, the DRL navigation system is divided into two categories: vision and laser.

[0012] Secondly, for laser DRL navigation systems that use laser sensors, the following steps are performed:

[0013] The adversarial map is defined as follows: a map in which the navigation agent's decisions near the modified location are unnecessarily altered by modifying obstacles in the original map, causing it to deviate from the original navigation path and ultimately fail to navigate.

[0014] For each original map, based on the fundamental principles of adversarial examples, the following steps are proposed to generate the corresponding adversarial map:

[0015] (1) Collect interaction information of the navigation agent: Perform multiple navigation tests on the DRL navigation model on the original map and record the interaction information between the navigation agent and the environment during each navigation process, including the mobile robot's posture information, sensor data, and actions performed.

[0016] (2) Based on the interaction information, identify security vulnerability areas:

[0017] (2a) Extract the observation information of the navigation agent from the interaction information as the original sample, use the adversarial sample generation algorithm to attack, record the attack results of each target action in each navigation and each time step, and add adversarial perturbation only to the sensor data during the attack.

[0018] (2b) Based on the recorded interaction information and the attack results of step (2a), a vulnerable area analysis map is generated; on the vulnerable area analysis map, several areas are selected from multiple areas where the attack was successful as the security vulnerable areas of the DRL navigation model on the original map; the selection rules are as follows: after a successful attack in this area, the action of the navigation agent will cause the mobile robot to deviate from the current navigation path.

[0019] (3) Calculate counter-disturbances based on the aforementioned vulnerable areas:

[0020] (3a) Let [a, b] represent the time step range of the security-vulnerable area, and then randomly select the observation information of a certain navigation from the interaction information collected in step (1) as the original sample, and use the JSMA algorithm to calculate the adversarial perturbation δ within the time step range [a, b].a δ a+1 ,...,δ b This allows the navigation agent to continuously perform target actions t after adding appropriate perturbations to the laser sensor data. Since the number of input features is relatively small, the calculation process of the feature map S in the JSMA algorithm is improved, as shown in the following formula:

[0021]

[0022]

[0023] Where, π t This represents the t-th output of the navigation agent's policy function π; k，i For time step k, the laser sensor data l k The i-th element, k∈[a, b]. S + (o k ,t) and S - (o k S and t) represent feature maps calculated by two different methods, respectively. + (o k In the t), only input features with partial derivatives greater than 0 are retained. In other words, as long as S is added... + (o k The input feature size of t)[i]>0 can make π t Get bigger, S + (o k The larger the value of t)[i], the more l is increased. k，i After numerical analysis, the DRL navigation model output is more inclined to reflect the target action t; similarly, S - (o k The larger the value of t)[i], the smaller l becomes. k，i The numerical model output is more inclined to the target action t; at each time step, the JSMA algorithm calculates the adversarial perturbation based on the corresponding feature map S;

[0024] (3b) Based on the countermeasures perturbation calculated in the previous step, calculate the general countermeasures perturbation δ using the following formula, which is the final countermeasures perturbation, where δ k，j Denotes the adversarial perturbation δ at the k-th time step. k The j-th component;

[0025]

[0026] (4) Implementing the perturbation based on the adversarial perturbation calculated in step 3: By modifying the position of obstacles near the vulnerable areas in the original map, the adversarial perturbation is added to the observation information of the navigation agent, that is, making the laser sensor data of the mobile robot in that area change to lk +δ, thus obtaining the adversarial map.

[0027] Preferably, step two is based on the idea of ​​adversarial training, which involves retraining the navigation agent by adding multiple adversarial maps to the training set, thereby enabling the navigation agent to navigate safely on various adversarial maps.

[0028] Preferably, step two specifically includes:

[0029] (1) Let M o Let M represent the set of original maps, where m ∈ M. o Multiple adversarial maps are generated using the adversarial map generation method described in step one, forming an adversarial map set. Finally, all map sets are obtained.

[0030] (2) For all maps m∈M, test the navigation success rate p of the navigation agent on that map. m ;

[0031] (3) Form a set of maps with a navigation success rate less than parameter p, i.e., training set M. E ={m|m∈M and p m <p}, retrain the navigation agent until the DRL navigation model forms a set M on the map. E Upper convergence;

[0032] (4) Repeat steps (2) and (3) of step two until for all maps m∈M, p m ≥p;

[0033] (5) Test the performance of the trained DRL navigation model.

[0034] Preferably, the navigation success rate is defined as the frequency with which the DRL navigation model successfully navigates multiple times on a map.

[0035] Preferably, the robustness of the DRL navigation model can be controlled by adjusting the parameter p through the security reinforcement training method.

[0036] Preferably, for a laser DRL navigation system using a laser sensor, the methods for modifying obstacles include modifying the shape of the obstacle, adding obstacles, and removing obstacles.

[0037] Preferably, for a visual navigation system, the way to modify obstacles includes pasting stickers at the location of the obstacle or changing the texture pattern.

[0038] Preferably, the observation information of the navigation agent includes the posture information of the mobile robot and sensor data.

[0039] The present invention also provides an application of the method in the field of artificial intelligence security technology.

[0040] (III) Beneficial Effects

[0041] This invention targets deep reinforcement learning (DRL) navigation applications. To improve the robustness of DRL navigation systems in real-world environments and ensure safe navigation on various maps, it proposes the definition and generation method of adversarial maps based on the fundamental principles of adversarial examples. Furthermore, by adopting an "attack-to-defense" approach, it designs a robustness enhancement method based on adversarial maps from the perspective of adversarial example attack and defense. This method addresses the problem of adding adversarial perturbations to the input of intelligent models in real-world environments, providing a usable solution for enhancing the robustness of DRL navigation models in real-world conditions. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a diagram illustrating the effectiveness of the anti-map attack method of this invention.

[0044] Figure 3 This is a diagram showing the navigation results of the model after security hardening according to the present invention.

[0045] Figure 4 The results show the navigation agent's navigation on the original map and the adversarial map. In each map, the solid circle in the lower right corner represents the target point, the upper left corner represents the starting point, and the curve represents the navigation path. Compared to the original map, the adversarial map only adds strip-shaped obstacles to locations outside the navigation path, causing the agent to change its initial navigation decisions, choosing a longer route, and ultimately failing due to navigation timeout.

[0046] Figure 5 This is a vulnerability region analysis diagram, where each sub-graph represents a different target action. The background is the original map, and each point represents the position of the agent at each time step in each navigation. Light-colored areas are areas that can be attacked as target actions, and dark-colored areas are areas where attacks have failed. Detailed Implementation

[0047] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0048] To improve the robustness of DRL navigation systems in real-world environments and ensure safe navigation on various maps, this invention proposes a definition and generation method for adversarial maps based on the fundamental principles of adversarial examples. Furthermore, it designs a robustness enhancement method based on adversarial maps from the perspective of adversarial example attack and defense, using an "attack-to-defense" approach.

[0049] The robustness enhancement method for deep reinforcement learning (DRL) navigation applications provided by this invention comprises two parts: 1) Generating adversarial maps: Based on the fundamental principles of adversarial examples, the definition and generation method of adversarial maps are proposed, while also pointing out the security vulnerabilities of DRL navigation systems; 2) Security-hardened training: The navigation agent is retrained using adversarial maps, thereby improving the robustness of the DRL navigation model. The following will refer to... Figure 1 Each part of the solution is described in detail, and the effectiveness of the invention is further illustrated by simulation experiments at the end.

[0050] (I) Methods for Generating Adversarial Maps

[0051] This section primarily defines adversarial maps and their generation methods, while also pointing out the security vulnerabilities of DRL navigation systems. The DRL navigation system comprises a DRL navigation model, sensors, and a mobile robot. The DRL navigation model refers to the DRL model used for navigation. Sensors are mounted on the mobile robot, and the DRL navigation model makes decisions based on sensor data, enabling the mobile robot to perform actions such as turning left or right. The mobile robot is a navigation agent, referring to a mobile robot that uses the DRL algorithm for navigation. Depending on the sensor used, DRL navigation systems can be divided into vision-based and laser-based systems. The definition of adversarial maps is given below, and the generation method of adversarial maps is described using a laser DRL navigation system with a laser sensor as an example.

[0052] The adversarial map proposed in this invention is defined as follows: a map in which minor modifications are made to obstacles in the original map (for laser DRL navigation systems, this includes modifying the shape of obstacles, adding obstacles at a certain location, removing obstacles at a certain location, etc.; for visual navigation systems, this includes pasting stickers at a certain location on obstacles, changing texture patterns, etc.), causing the navigation agent to make unnecessary changes in its decisions near the modified location, deviating from the original navigation path, and ultimately failing to navigate.

[0053] For each original map, this section of the solution, based on the fundamental principles of adversarial examples, provides the following steps to generate the corresponding adversarial map:

[0054] (1) Collect interaction information of the navigation agent: Perform multiple navigation tests on the DRL navigation model on the original map and record the interaction information between the navigation agent and the environment during each navigation process, including the mobile robot's posture information, sensor data, and actions performed.

[0055] (2) Based on the interaction information, identify security vulnerability areas:

[0056] (2a) Extract the observation information (including the posture information of the mobile robot and sensor data) of the navigation agent from the interaction information as the original sample, use the adversarial sample generation algorithm to attack, record the attack result (success or failure) of each target action (forward, left turn, right turn) in each navigation and each time step, and only add adversarial perturbation to the sensor data during the attack.

[0057] (2b) Based on the recorded interaction information and the attack results of step (2a), a vulnerable area analysis map is generated; on the vulnerable area analysis map, several areas are selected from the multiple areas where the attack was successful as the security vulnerable areas of the DRL navigation model on the original map; the selection rules are as follows: after the attack is successful in the area, the action of the navigation agent will cause the mobile robot to deviate from the current navigation path.

[0058] (3) Calculate counter-disturbances based on the aforementioned vulnerable areas:

[0059] (3a) Let [a, b] represent the time step range of the security-vulnerable area, and then randomly select the observation information of a certain navigation from the interaction information collected in step (1) as the original sample, and use the JSMA algorithm to calculate the adversarial perturbation δ within the time step range [a, b]. a δ a+1 ,...,δ b This allows the navigation agent to continuously perform target actions t after adding appropriate perturbations to the laser sensor data. Since the number of input features is relatively small, the calculation process of the feature map S in the JSMA algorithm is appropriately improved. The detailed formula is as follows:

[0060]

[0061]

[0062] Where, π t This represents the t-th output of the navigation agent's policy function π; k，i For time step k, the laser sensor data l k The i-th element, k∈[a, b]. S + (o k ,t) and S - (o kS and t) represent feature maps calculated by two different methods, respectively. + (o k In the t), only input features with partial derivatives greater than 0 are retained. In other words, as long as S is added... + (0 k The input feature size of t)[i]>0 can make π t Get bigger, S + (o k The larger the value of t)[i], the more l is increased. k，i After numerical analysis, the DRL navigation model output is more inclined to reflect the target action t. Similarly, S - (o k The larger the value of t)[i], the smaller l becomes. k，i The numerical model output is more aligned with the target action t. At each time step, the JSMA algorithm calculates adversarial perturbations based on the corresponding feature map S.

[0063] (3b) Based on the countermeasures perturbation calculated in the previous step, calculate the general countermeasures perturbation δ using the following formula, which is the final countermeasures perturbation. Wherein, δ k，j Denotes the adversarial perturbation δ at the k-th time step. k The j-th component.

[0064]

[0065] (4) Implementing the perturbation based on the adversarial perturbation calculated in step 3: By modifying the position of obstacles near the vulnerable areas in the original map, the adversarial perturbation is added to the observation information of the navigation agent, that is, the laser sensor data of the mobile robot in this area becomes l k +δ, for laser DRL navigation systems, can be achieved by modifying the shape of obstacles, adding obstacles at a certain location, or removing obstacles at a certain location.

[0066] The above process is described using DRL laser navigation as an example. The process still applies to visual navigation, with the only difference being in calculating counter-disturbances.

[0067] (II) Safety Reinforcement Training

[0068] This section proposes a security-hardened training method. Based on the idea of ​​adversarial training, this method retrains the navigation agent by adding multiple adversarial maps to the training set, enabling it to navigate safely on various adversarial maps and ultimately improving the robustness of the DRL navigation model. This security-hardened training method consists of the following steps:

[0069] (1) Let M o Let M represent the set of original maps, where m ∈ M. oMultiple adversarial maps are generated using the adversarial map generation method proposed in this invention, forming an adversarial map set. Finally, all map sets are obtained.

[0070] (2) For all maps m∈M, test the navigation success rate p of the navigation agent on that map. m ;

[0071] (3) Form a set (training set) M of maps with a navigation success rate less than p. E ={m|m∈M and p m <p}, retrain the navigation agent until the DRL navigation model forms a set M on the map. E Upper convergence;

[0072] (4) Repeat steps (2) to (3) until for all maps m∈M, p m ≥p;

[0073] (5) Test the performance of the DRL navigation model after reinforcement training.

[0074] The navigation success rate is defined as the frequency with which the DRL navigation model successfully navigates on a map during multiple tests. This security-hardening training method controls the robustness of the DRL navigation model by adjusting the parameter p. A larger p results in higher robustness of the hardened DRL navigation model, but also makes it more difficult for the model to converge during training.

[0075] (III) Simulation Experiment

[0076] The effects of the present invention will be further explained below with reference to simulation experiments.

[0077] (1) Simulation Experiment Conditions

[0078] In this invention, ROS (Robot Operating System) is used as the development system for the mobile robot; the mobile robot model is Turtlebot3, equipped with a single-line laser rangefinder. The operating system is Ubuntu 20.04 LTS, and the graphics card is an NVIDIA GeForce GTX 1080Ti.

[0079] (2) Training the navigation agent

[0080] This invention uses the Gazebo software in ROS to provide physical simulation capabilities, builds a reinforcement learning environment, and creates several original maps to train the navigation agent until the DRL navigation model achieves a navigation success rate of over 95% on all maps. The navigation agent is implemented using the PPO algorithm, and both the Actor network and the Critic network consist of three fully connected layers.

[0081] (3) Simulation Experiment Process and Results

[0082] For each original map, two types of adversarial maps are first generated: one type implemented by removing obstacles, and the other type implemented by adding obstacles. In addition, a control map is generated by randomly adding and removing obstacles to demonstrate that the initially trained DRL navigation model has a certain generalization ability and robustness, highlighting the significance of this invention. Then, the navigation agent's navigation success rate is tested on all maps to obtain... Figure 2 .

[0083] Finally, the DRL navigation model was security-hardened and trained using all maps, including adversarial maps and comparative maps, with ρ=90% during training; the hardened DRL navigation model was then tested to obtain... Figure 3 .

[0084] from Figure 2 It can be seen that when navigating on an adversarial map, the navigation agent makes unnecessary changes to its decisions near areas of vulnerability, ultimately leading to navigation failure; while it can still navigate normally on the control group map. Let's look at... Figure 3 After reinforcement training, the navigation agent achieved a navigation success rate of over 90% on all maps, significantly higher than before reinforcement. Experimental results demonstrate that the adversarial map proposed in this invention has a significant attack effect, providing a research direction for improving the robustness of DRL navigation models. The security reinforcement training method proposed in this invention enables the navigation agent to navigate safely on adversarial maps without affecting the original performance, thus improving the robustness of DRL navigation models.

[0085] As can be seen, this invention, for the first time, proposes a robustness enhancement method for DRL navigation from the perspective of adversarial attack and defense under real-world conditions. Its basic idea is to study adversarial attack methods for DRL navigation under real-world conditions to uncover the security vulnerabilities of the DRL navigation model; then, based on the idea of ​​adversarial training, conduct corresponding security hardening research to ultimately improve the robustness of the DRL navigation model.

[0086] Compared with existing methods, the present invention has the following advantages:

[0087] (1) In the adversarial map generation method, the adversarial map attack proposed in this invention is based on the basic principle of adversarial samples, which solves the problem of how to add adversarial disturbances to the input of intelligent models in real environment, reveals the security risks of DRL navigation models in practical applications, and provides new ideas for the practical application of artificial intelligence attack and defense research.

[0088] (2) In the security reinforcement training method, the present invention reinforces the DRL navigation model based on adversarial maps, providing a usable solution for enhancing the robustness of the DRL navigation model in real environment.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A robustness enhancement method for deep reinforcement learning navigation applications based on adversarial maps, characterized in that, Includes the following steps: Step 1: Based on the fundamental principles of adversarial examples, define adversarial maps and propose adversarial map generation methods; Step 2, Security Hardening Training: Using the adversarial map generation method from Step 1, the navigation agent is retrained by generating an adversarial map; In step one, the DRL navigation system is first defined as including a DRL navigation model, sensors, and a mobile robot. The DRL navigation model refers to the DRL model used for navigation. The sensors are mounted on the mobile robot. The DRL navigation model can make decisions based on the sensor data, enabling the mobile robot to perform various actions. The mobile robot is a navigation agent, referring to a mobile robot that uses the DRL algorithm for navigation. Depending on the type of sensor, the DRL navigation system is divided into two categories: vision and laser. Secondly, for laser DRL navigation systems that use laser sensors, the following steps are performed: The adversarial map is defined as follows: a map in which the navigation agent's decisions near the modified location are unnecessarily altered by modifying obstacles in the original map, causing it to deviate from the original navigation path and ultimately fail to navigate. For each original map, based on the fundamental principles of adversarial examples, the following steps are proposed to generate the corresponding adversarial map: (1) Collect interaction information of the navigation agent: Perform multiple navigation tests on the DRL navigation model on the original map and record the interaction information between the navigation agent and the environment during each navigation process, including the mobile robot's posture information, sensor data, and actions performed. (2) Based on the interaction information, identify security vulnerability areas: (2a) Extract the observation information of the navigation agent from the interaction information as the original sample, use the adversarial sample generation algorithm to attack, record the attack results of each target action in each navigation and each time step, and add adversarial perturbation only to the sensor data during the attack. (2b) Based on the recorded interaction information and the attack results of step (2a), a vulnerable area analysis map is generated; on the vulnerable area analysis map, several areas are selected from multiple areas where the attack was successful as the security vulnerable areas of the DRL navigation model on the original map; the selection rules are as follows: after a successful attack in this area, the action of the navigation agent will cause the mobile robot to deviate from the current navigation path. (3) Calculate counter-disturbances based on the aforementioned vulnerable areas: (3a) Let [a,b] represent the time step range of the security vulnerability area, and then randomly select the observation information of a certain navigation from the interaction information collected in step (1) as the original sample, and use the JSMA algorithm to calculate the adversarial perturbation δ within the time step range [a,b]. a ,δ a+1 ,…,δ b This allows the navigation agent to continuously perform target actions t after adding appropriate perturbations to the laser sensor data. Since the number of input features is relatively small, the calculation process of the feature map S in the JSMA algorithm is improved, as shown in the following formula: Where, π t This represents the t-th output of the navigation agent's policy function π; k,i For time step k, the laser sensor data l k The i-th element, k∈[a,b]; S + (o k ,t) and S - (o k S and t represent feature maps calculated by two different methods, respectively. + (o k In the ,t) input, only the input features with partial derivatives greater than 0 are retained. In other words, as long as S is added... + (o k Given the input feature size of t)[i]>0, we can make π t Get bigger, S + (o k The larger the value of t)[i], the more l is increased. k,i After numerical analysis, the DRL navigation model output is more inclined to reflect the target action t; similarly, S - (o k The larger the value of t)[i], the smaller l becomes. k,i The numerical model output is more inclined to the target action t; at each time step, the JSMA algorithm calculates the adversarial perturbation based on the corresponding feature map S; (3b) Based on the countermeasures perturbation calculated in the previous step, calculate the general countermeasures perturbation δ using the following formula, which is the final countermeasures perturbation, where δ k,j Denotes the adversarial perturbation δ at the k-th time step. k The j-th component; (4) Implementing the perturbation based on the adversarial perturbation calculated in step 3: By modifying the position of obstacles near the vulnerable areas in the original map, the adversarial perturbation is added to the observation information of the navigation agent, that is, making the laser sensor data of the mobile robot in that area change to l k +δ, thus obtaining the adversarial map; Step 2 is based on the idea of ​​adversarial training. By adding multiple adversarial maps to the training set, the navigation agent is retrained, thereby enabling the navigation agent to navigate safely on various adversarial maps. Step two specifically includes: (1) Let M o Let M represent the set of original maps, where m ∈ M. o Multiple adversarial maps are generated using the adversarial map generation method described in step one, forming an adversarial map set. Finally, all map sets are obtained. (2) For all maps m∈M, test the navigation success rate p of the navigation agent on that map. m ; (3) Form a set of maps with a navigation success rate less than the parameter p, that is, the training set M E ={m|m ∈ M and p m < p}, retrain the navigation agent until the DRL navigation model converges on the map formation set M E ; (4) Repeat steps (2) and (3) of step two until for all maps m∈M, p m ≥p; The performance of the trained DRL navigation model was tested; The security hardening training method controls the robustness of the DRL navigation model by adjusting the parameter p.

2. The method as described in claim 1, characterized in that, The navigation success rate is defined as the frequency with which the DRL navigation model successfully navigates on a map when tested multiple times.

3. The method as described in claim 1, characterized in that, For laser DRL navigation systems that use laser sensors, obstacles can be modified by changing their shape, adding obstacles, or removing obstacles.

4. The method as described in claim 1, characterized in that, For visual navigation systems, ways to modify obstacles include pasting stickers at the obstacle's location or changing the texture pattern.

5. The method as described in claim 1, characterized in that, The observation information of the navigation agent includes the mobile robot's posture information and sensor data.

6. The application of the method as described in any one of claims 1 to 5 in the field of artificial intelligence security technology.

Images