Robot navigation method based on reinforcement learning safety constraint and danger perception

Abstract

The invention discloses a robot navigation method based on reinforcement learning security constraint and danger perception, and relates to the technical field of robots and artificial intelligence, and the robot navigation method comprises the following steps: when a robot is controlled to execute a navigation task in a target area, using a security filter SF to perform security constraint on the execution action of the robot; and when the robot executes the navigation task, the danger perception virtual robot HAVR is utilized to obtain experience in the dangerous area so as to enhance the ability of the robot to explore in the dangerous area. The action generated by reinforcement learning can be dynamically adjusted, so that the action meets the safety constraint, and the safety of the robot in the reinforcement learning training process is further enhanced; by introducing the HAVR with risk-driven rewards, the robot can still obtain experience in a dangerous area while introducing security constraints, and the exploration security of the robot is further enhanced while the performance of the model after convergence is ensured.

Description

Technical Field

[0001] This invention relates to the fields of robotics and artificial intelligence, and more specifically, to a robot navigation method based on reinforcement learning safety constraints and hazard perception. Background Technology

[0002] With the rapid development of robotics and artificial intelligence, autonomous mobile robots have been applied in various complex real-world environments involving dynamic crowds. They are expected to reach their destinations safely, efficiently, and politely, avoiding both overly aggressive approaches that could intrude on human paths and cause collisions, and overly conservative approaches that hinder social navigation. To achieve this goal, designing safe and efficient social navigation algorithms is crucial. Current robot navigation algorithms primarily employ deep reinforcement learning-based methods.

[0003] In existing technologies, learning based on the trial-and-error mechanism of reinforcement learning often leads to robots performing unsafe actions during training, posing risks to their exploration. Furthermore, some methods apply safety constraints throughout the entire reinforcement learning training process, but limit the robot's ability to gain experience in high-risk areas, thus affecting the performance of the trained model to some extent. Additionally, while methods that do not add safety layers in the early stages of reinforcement learning training effectively alleviate the problem of limited robot exploration, they also lead to reduced safety in the early exploration phase. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a robot navigation method based on reinforcement learning safety constraints and hazard perception, which enhances robot exploration safety while ensuring performance after model convergence.

[0005] To achieve the above technical objectives, this application provides a robot navigation method based on reinforcement learning safety constraints and hazard perception, comprising the following steps: In response to controlling the robot to perform navigation tasks within the target area, a safety filter SF is used to impose safety constraints on the robot's actions. In response to the robot performing navigation tasks, the Hazard Awareness Virtual Robot (HAVR) is used to gain experience in hazardous areas, thereby enhancing the robot's ability to explore hazardous areas.

[0006] Preferably, when controlling the robot to perform navigation tasks, the social navigation performed by the robot is modeled as a partially observable Markov decision process, which consists of the agent's state space, action space, transition probability, reward function, observation space, observation probability, and discount factor.

[0007] Preferably, when using the safety filter SF for safety constraints, the safety filter SF is generated by using model predictive control (MPC) and dynamic control barrier function (D-CBF).

[0008] Preferably, when gaining experience in hazardous areas, the introduction of HAVR with risk-driven rewards enables the robot to gain experience in hazardous areas even while introducing SF.

[0009] Preferably, when gaining experience in dangerous areas, Risk Exploration Reinforcement Learning (HERL) is used to strategically incentivize the HAVR to explore high-risk areas. Positive rewards are assigned when the HAVR approaches or collides with a pedestrian to guide the HAVR in instructional interactions with the pedestrian. The mathematical expression of the reward function is defined as follows: In the formula, express The closest distance between the robot and the pedestrian. express The distance between the robot and the target point at any given time.

[0010] Preferably, when using robots and HAVR to explore and gain experience, a dual experience pool equipped with a balanced sampling mechanism is used to maintain the balance of the distribution of the two types of experience.

[0011] Preferably, when gaining experience in hazardous areas, the experience pool used to train SFRL is divided into two parts, storing the robot's interaction experience with pedestrians and the HAVR's exploration experience, respectively. During training, empirical data is sampled equally from both partitions.

[0012] Based on the same inventive concept, this invention discloses a robot navigation system based on reinforcement learning safety constraints and hazard perception, the system comprising: The safety constraint module is used to constrain the robot's actions by employing the safety filter SF when controlling the robot to perform navigation tasks within the target area. The Hazard Learning Module is used to enhance the robot's ability to explore hazardous areas by leveraging the Hazard Awareness Virtual Robot (HAVR) to gain experience in hazardous areas while the robot is performing navigation tasks.

[0013] The present invention discloses the following technical effects: This invention proposes the HAESRL algorithm by combining SF and HAVR. This algorithm introduces safety constraints to make safety corrections to the actions generated by reinforcement learning, while introducing HAVR to ensure that the robot acquires experience in dangerous areas. This enhances the safety of robot exploration and ensures the performance of the model after convergence.

[0014] During reinforcement learning training, due to its trial-and-error learning nature, strategies may generate unsafe actions that could lead to collisions between the robot and pedestrians, especially in the early stages of training. To address this issue, SF (Safety Default) is designed, combining MPC (Multi-Process Control) and D-CBF (Dual-Process Default). SF can dynamically adjust the actions generated by reinforcement learning to meet safety constraints, thereby enhancing the robot's safety during reinforcement learning training.

[0015] In reinforcement learning training, introducing safety constraints (SF) can effectively reduce the probability of a robot entering dangerous areas, thereby enhancing its safety during exploration. However, this approach leads to a lack of experiential data on dangerous areas, reducing data diversity and ultimately weakening the performance of the trained reinforcement learning policy, especially in terms of navigation safety. To address this issue, risk-driven reward-based high-risk dynamic learning (HAVR) is introduced, enabling the robot to gain experience in dangerous areas while incorporating safety constraints (SF). This ensures performance after model convergence while further enhancing the robot's exploration safety. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the method described in this invention; Figure 2 This is the principle behind the invention that helps robots acquire experience on better paths; Figure 3 This is the simulation environment described in this invention; Figure 4 These are the collision rate training curves of the algorithms described in this invention; Figure 5 These are the navigation trajectories of different algorithms described in this invention under the same social navigation scenario. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] like Figures 1-5 As shown, this invention provides a robot navigation method based on reinforcement learning safety constraints and hazard perception exploration. By combining a safety filter (SF) and a hazard-aware virtual robot (HAVR), a hazard-aware exploration-based safety reinforcement learning (HAESRL) algorithm is designed. Its overall framework diagram is shown below. Figure 1 As shown, the framework mainly consists of three modules: robot exploration, HAVR exploration, and SF. SF ensures that the robot's actions meet safety constraints; HAVR enables the robot to gain experience in hazardous areas. Specifically, Safety-Filtered Reinforcement Learning (SFRL) represents RL used in conjunction with safety filtering mechanisms; Hazard Exploration Reinforcement Learning (HERL) represents RL for exploring hazardous areas; and the virtual environment is a replica of the environment constructed by the robot based on its own perception.

[0020] 1. Robot navigation (the robot can be a social robot or something else): A typical scenario for robot navigation involves an autonomously moving robot and several freely walking pedestrians. The robot needs to navigate from a starting point to a destination, avoiding dynamic crowds and maintaining appropriate social distancing while reaching the destination as efficiently as possible to achieve safe and efficient robot navigation. To achieve this goal, social navigation can be formalized as a sequential decision problem. This invention models it as a Partially Observable Markov Decision Process (POMDP), which is typically defined as a seven-tuple. .in This represents the state space of the intelligent agent. Represents the action space. Represents the transition probability. It is a reward function. It is the observation space. It is the probability of observation. This is the discount factor. By elaborating on these fundamental concepts, the following section presents a formalized expression for reinforcement learning in the robot navigation problem.

[0021] (1) State space: In robot navigation scenarios, the state space at each time step consists of two parts: one part is the observable state that can be observed by other agents, including the agent's position. ,speed and radius The other part is the unobservable state that can only be perceived by the agent itself, including the agent's target position. Preferred speed and deflection angle This invention employs a robot-centric coordinate system, making the state representation more universal. After this transformation, the global state used for social navigation... It can be represented as: (1) in, express The robot's status at all times. express Pedestrians at all times state, This indicates the distance between the robot and the target point. Representing robots and pedestrians The distance between them and They represent pedestrians. The relative position and relative speed of the robot.

[0022] (2) Action space: This invention uses continuous motion to control the robot's movement. Specifically, the robot at a given moment... The actions generated by reinforcement learning are defined as follows: (2).

[0023] (3) Reward function: In social navigation tasks, adhering to human social norms is crucial for robots. To achieve this, this invention designs a reward function to encourage the robot to approach the target and complete the task, while penalizing collisions or uncomfortable social distancing. Its mathematical form is as follows: (3) in, express The closest distance between the robot and the pedestrian. express The distance between the robot and the target point at any given time.

[0024] (4) Optimization objective: The objective of the agent is to determine the optimal policy to maximize the state-action value function. : (4); Defined by the Bellman optimal equation: (5).

[0025] 2. Safety filter: This invention designs SF by combining Model Predictive Control (MPC) with Dynamic Control Barrier Function (D-CBF). To facilitate D-CBF design, the robot position, obstacle position, and obstacle shape are defined as follows: , and And define variables. To construct the MPC, this invention describes the robot's dynamic model using discrete-time equations: (6) in, Indicates the system at time step The state at that time, Indicates time step The control input for the time, here, , .

[0026] During reinforcement learning training, policies may generate unsafe actions that could lead to collisions between the robot and pedestrians, especially in the early stages of training. To address this issue, this section designs SF by combining MPC with D-CBF. SF can dynamically adjust the actions generated by reinforcement learning to ensure that safety constraints are met, thereby enhancing safety during reinforcement learning training.

[0027] For safety-critical control issues, safety clusters Defined as a continuously differentiable function Superlevel set: (7) Definition 1: (Discrete-time D-CBF). Consider a discrete-time system. Given a function... Defined collection If there exists a function If the control system satisfies the following conditions, then it is called a function. For discrete D-CBF: (8) in, To simplify processing, the function will be... Choose as a constant .

[0028] At time step At that time, the set of all moving obstacles is defined as and obstacles In the time domain The forward prediction state is denoted as To simplify the expression, use replace ,make This invention employs a constant velocity model for trajectory prediction. At time step... season , The robot's radius is Moving obstacles The radius is Therefore, D-CBF can be expressed in the following quadratic form: (9).

[0029] This invention combines MPC with D-CBF, and the design problem of SF can be described as follows: (10) in, Indicates at time step Initial conditions at time , and These represent the feasible states, inputs, and terminal states, respectively.

[0030] 3. Danger-sensing virtual robot: In reinforcement learning training, introducing risk-driven learning (SF) can effectively reduce the probability of a robot entering dangerous areas, thereby enhancing the robot's safety during exploration. However, this method leads to a lack of experiential data on dangerous areas, reducing data diversity and ultimately weakening the performance of the trained reinforcement learning strategy, especially in terms of navigation safety. To address this issue, this invention introduces a risk-driven reward system (HAVR) that allows the robot to gain experience in dangerous areas while incorporating SF. To promote experiential learning in dangerous areas, especially adverse encounters that provide guiding feedback, this invention proposes a risk-driven learning model (HERL) that strategically incentivizes HAVR to explore high-risk areas. The core of HERL training is a risk-driven reward mechanism that guides HAVR to engage in instructional interactions with pedestrians by assigning positive rewards when HAVR approaches or collides with a pedestrian. The mathematical definition of this reward function is as follows: (11) At each time step, HAVR employs a two-stage simulation process to help the robot acquire experience in hazardous areas. HAVR first executes the actions generated by HERL. Collect experience data Then execute the action generated by SFRL. Collect experience data This method can help robots gain experience on better paths, and the principle is as follows: Figure 2 As shown in the diagram, the orange area represents a danger zone. The black line represents the robot's exploration trajectory under SF (Self-Protected Path), the green line represents a better path, and the red line represents the exploration trajectory under HAVR (Habit-Assisted Path). By introducing HAVR for exploration of danger zones, the robot can gain additional experience data, thereby improving the generalization ability of its strategy and its environmental adaptability. Furthermore, this method accelerates the acquisition of adverse experiences by the robot, allowing it to learn from them and further enhance safety during the exploration process.

[0031] To effectively utilize the experience gained from robot and HAVR exploration, this invention employs a dual experience pool equipped with an equal sampling mechanism to maintain the balance of the two types of experience distributions. Specifically, the experience pool used for training SFRL is equally divided into two parts, storing robot-pedestrian interaction experience and HAVR exploration experience, respectively. During training, experience data is sampled equally from both partitions. This method ensures that the model can optimize its strategy based on both robot-pedestrian interaction experience and the experience provided by HAVR.

[0032] The complete flowchart of the HAESRL algorithm is shown in Table 1.

[0033] Table 1 The following is a detailed simulation experiment demonstrating the effectiveness of the proposed HAESRL algorithm in exploring security and social navigation performance.

[0034] 4. Simulation Experiment: 4.1 Simulation Environment: The simulation environment for this experiment is the CrowdNav social-aware navigation simulation platform based on Python, running on Ubuntu 20.04. For example... Figure 3 As shown, white circles represent robots, circles of different colors represent different pedestrians, pentagrams represent the robot's target points, and triangles represent the targets of each pedestrian. The radii of both the robot and pedestrians are set to 0.3 meters. In each simulation round, the robot's start and end points are set to (0, -4) and (0, 4), respectively, while the pedestrians' start and end points are randomly distributed near the edge of a circle with a radius of 4 meters. At each time step, the robot selects its action based on its local observations and current strategy, while the pedestrians' actions are controlled by the ORCA algorithm.

[0035] 4.2 Baseline: To evaluate the performance of the proposed algorithm, this invention selects several existing state-of-the-art methods as baselines for comparison, including SARL, CBF-RL, SSA-RL, and USL. SARL is an unconstrained socially aware navigation algorithm, while CBF-RL, SSA-RL, and USL are three existing state-of-the-art safety reinforcement learning algorithms. To ensure fairness in the comparison, HAESRL and the other three safety reinforcement algorithms are all built upon the SARL algorithm, and each introduces its own safety mechanism. Except for the differences in the safety mechanisms, all other experimental settings remain consistent.

[0036] 4.3 Evaluation Indicators: To quantitatively evaluate the performance of different algorithms, this invention selects the following representative indicators for comparative analysis: Success Rate (SR): The proportion of times the robot reaches the target point without collision within the maximum navigation time; Collision Rate (CR): The ratio of collisions between the robot and pedestrians; Average Navigation Time (ANT): The average time required for the robot to reach the target location; Average Discounted Reward (ADR): The average cumulative discounted reward obtained by the robot; Discomfort Frequency (DF): The percentage of time the robot's closest distance to a pedestrian is less than 0.2 meters. Collision Rate Training Curve: A curve describing the change in collision rate with the number of training rounds.

[0037] 4.4 Parameter Settings: In the comparative experiments, all algorithms were trained under the same environmental hyperparameters and used the same value network structure, policy network structure, and parameter settings. The HAESRL algorithm proposed in this invention is implemented in PyTorch and trained using the Adam optimizer. The specific hyperparameter settings are shown in Table 2.

[0038] Table 2 Hyperparameter Settings 4.5 Quantitative Analysis: After 50,000 training rounds, the collision rate training curves for each algorithm are as follows: Figure 4 As shown in the figure, the results indicate that the HAESRL algorithm maintains a low collision rate throughout the entire training process, significantly outperforming the baseline algorithms. CBF-RL and SSA-RL show good performance in the early stages of training, but their collision rates are even higher than SARL in the later stages. USL performs poorly due to the lack of a safety layer mechanism in the early stages of training. The above analysis shows that HAESRL effectively enhances the safety of the original method during the exploration process and outperforms CBF-RL, SSA-RL, and USL.

[0039] Based on the trained model, test environments with different numbers of pedestrians (5, 7, and 9 people) were randomly selected, and the algorithm performance was further verified through quantitative analysis. No safety mechanisms were enabled for any of the safety reinforcement learning methods during the testing phase (e.g., HAESRL did not use SF). Evaluation metrics included SR, CR, ANT, ADR, and DF. To ensure consistency and fairness, each algorithm underwent 500 rounds of testing, and the results are shown in Table 3. Experimental results show that in any test environment, HAESRL's success rate is higher than the baseline algorithm, while its collision rate and adverse frequency are lower. Its average navigation time is shorter than CBF-RL and SSA-RL, but longer than SARL and USL. In terms of average discounted reward, HAESRL consistently outperforms CBF-RL, SSA-RL, and USL. In a simpler scenario with 5 pedestrians, HAESRL's average discounted reward is lower than SARL's, due to its longer average navigation time when other metrics are similar to SARL. As the environment becomes more complex (7 and 9 people), its average discounted reward surpasses SARL, demonstrating its advantage in complex scenarios. The above analysis shows that HAESRL improves the safety of the original method while also making the strategy more conservative. Furthermore, in maintaining the socially aware navigation performance of the original method, this algorithm outperforms other safety reinforcement learning baseline algorithms. Notably, as environmental complexity increases, HAESRL exhibits more stable performance than baseline algorithms, indicating its superior generalization ability.

[0040] Table 3 Quantitative Assessment Results 4.6 Qualitative Analysis: To further evaluate the social navigation performance of the proposed algorithm, this section plots the robot's navigation trajectories under different methods in the same randomly generated environment and performs qualitative analysis. Relevant details are as follows... Figure 5 As shown. The yellow disk represents the robot, the red star represents the robot's target position, and the circle represents the pedestrian. The numbers on the disk and circle represent the time of arrival at the corresponding position. The colored curves represent the corresponding trajectories of the robot and the pedestrian. At approximately 7 seconds, the robot using SARL chooses to move to the lower right to avoid a pedestrian approaching from its lower left, but collides with a pedestrian approaching from behind at 8 seconds. The robot using CBF-RL avoids pedestrians approaching from its left by first moving right and then left in the initial navigation phase. In addition, at approximately 4 seconds, the robot begins a deceleration lasting about 3 seconds, even though it is still far from the pedestrians in front of it. These behaviors result in a longer navigation time. The robot using SSA-RL also avoids pedestrians approaching from the left by first moving right and then left in the initial navigation phase, and then retreats at approximately 6 seconds to avoid pedestrians approaching rapidly from the right. These behaviors also result in a longer navigation time. Around the 8-second mark, the robot using USL, facing pedestrians approaching from both the lower left and lower right, swayed left and right, ultimately colliding with the pedestrian on the left at the 12-second mark. In contrast, the robot using HAESRL avoided the pedestrian approaching from the left by first slightly shifting to the right and then circling around, and then avoided the pedestrian rapidly approaching from the right by circling to the left around the 6-second mark, ultimately reaching the target position in a shorter time. These results demonstrate that HAESRL can help the original method find a safe trajectory, and that the navigation time for this safe trajectory is shorter than that of CBF-RL and SSA-RL.

[0041] Simulation results show that the proposed HAESRL algorithm significantly improves the exploration safety of the original method while ensuring the performance after model convergence. Compared with existing state-of-the-art secure reinforcement learning methods, HAESRL achieves optimal performance.

[0042] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0044] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A robot navigation method based on reinforcement learning safety constraints and hazard perception, characterized in that, Includes the following steps: In response to controlling the robot to perform navigation tasks within the target area, a safety filter SF is used to impose safety constraints on the robot's actions. In response to the robot performing navigation tasks, the Hazard Awareness Virtual Robot (HAVR) is used to gain experience in hazardous areas, thereby enhancing the robot's ability to explore hazardous areas.

2. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 1, characterized in that: When controlling a robot to perform navigation tasks, the social navigation performed by the robot is modeled as a partially observable Markov decision process, which consists of the agent's state space, action space, transition probabilities, reward function, observation space, observation probabilities, and discount factor.

3. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 2, characterized in that: When using the safety filter SF for safety constraints, the safety filter SF is generated by model predictive control (MPC) and dynamic control barrier function (D-CBF).

4. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 3, characterized in that: When gaining experience in hazardous areas, the robot can still gain experience in hazardous areas while introducing risk-driven rewards (HAVR).

5. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 4, characterized in that: When gaining experience in dangerous areas, Risk Exploration Reinforcement Learning (HERL) is used to strategically incentivize HAVR to explore high-risk areas. Positive rewards are assigned when HAVR approaches or collides with pedestrians to guide HAVR in instructional interactions with pedestrians. The mathematical definition of the reward function is as follows: In the formula, express The closest distance between the robot and the pedestrian. express The distance between the robot and the target point at any given time.

6. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 5, characterized in that: When using robots and HAVR to explore and gain experience, a dual experience pool equipped with a balanced sampling mechanism is used to maintain the balance of the distribution of the two types of experience.

7. The robot navigation method based on reinforcement learning safety constraints and hazard perception according to claim 6, characterized in that: When gaining experience in hazardous areas, the experience pool used to train SFRL is divided into two parts, storing robot-pedestrian interaction experience and HAVR exploration experience respectively. During training, empirical data is sampled equally from both partitions.

8. A robot navigation system based on reinforcement learning safety constraints and hazard perception, used to implement the robot navigation method based on reinforcement learning safety constraints and hazard perception as described in claim 1, characterized in that, The system includes: The safety constraint module is used to apply safety constraints to the robot's actions when controlling the robot to perform navigation tasks within a target area, using a safety filter SF. The Hazard Learning Module is used to enhance the robot's ability to explore hazardous areas by leveraging the Hazard Awareness Virtual Robot (HAVR) to gain experience in hazardous areas while the robot is performing navigation tasks.

Images