Multi-robot safe navigation method and system based on hierarchical deep reinforcement learning

Abstract

The application discloses a multi-robot safe navigation method and system based on layered deep reinforcement learning, belongs to the technical field of deep reinforcement learning, and particularly relates to a multi-robot navigation method based on deep reinforcement learning. The method adopts a layered deep reinforcement learning model framework based on skills, designs a rule-based target driving strategy and a safe turning strategy, and a barrier-avoiding navigation strategy based on deep reinforcement learning, and realizes three low-layer behavior strategies of target approaching, emergency braking and barrier-avoiding navigation respectively; a high-layer behavior selection strategy is trained through deep reinforcement learning, and a good behavior selection model can effectively avoid the dependence on the selection rules of the artificially designed strategy. The application strengthens the recognition ability of the robot to the current environment and improves the rapidity and safety of the barrier-avoiding navigation of the multi-robot by designing multiple sub-behavior strategies based on traditional control rules and introducing a behavior selection strategy based on deep reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning technology, specifically to a multi-robot safe navigation method and system based on hierarchical deep reinforcement learning. Background Technology

[0002] With the development of robotics and electronic information technology, mobile robots are becoming increasingly intelligent, effectively improving human productivity while also replacing humans in dangerous tasks. Navigation is the foundation for all mobile robot work, requiring the robot to move from a starting point to a target point without collision in a short time. Multi-robot systems, building upon this foundation, need to handle more complex scenarios and possess stronger dynamic characteristics. Robots must not only avoid external obstacles but also coordinate with each other without colliding. Therefore, the navigation problem in multi-robot systems places higher demands on the robots' obstacle avoidance and collaborative capabilities.

[0003] Traditional multi-robot navigation problems can be divided into centralized and distributed approaches. Centralized methods assume that a central server can access the global map and obtain the real-time status of all robots through a global communication system. Based on this information, the central server can plan the optimal trajectory for each robot. However, due to limitations in the central server's computing power and reliable communication capabilities, centralized methods are difficult to scale to large-scale multi-robot systems. Distributed methods assume that each robot can obtain the status of other robots and make independent decisions. Most distributed methods are based on velocity barrier (VO) frameworks, such as RVO, ORCA, and NH-ORCA. These methods all assume that each robot has perfect perception of its surroundings. However, in practical applications, sensors have errors, and communication networks suffer from latency and packet loss. In addition, these methods have many adjustable parameters that are sensitive to the environment, which also limits their applicability.

[0004] With the development of deep reinforcement learning technology, multi-robot navigation methods based on deep reinforcement learning have emerged continuously. In 2016, Chen et al. first proposed an agent-level multi-robot obstacle avoidance algorithm (CADRL) based on deep reinforcement learning. The CADRL algorithm transforms the large amount of online computation in traditional methods into offline training of value networks. The robot can obtain its own and its neighbors' states and directly obtain collision-free action instructions through the value network. However, the states of the robot's neighbors cannot be directly obtained, and the training process is very time-consuming, which limits the scope of application of this algorithm. In order to simplify the robot's behavior decision-making process, sensor-level multi-robot navigation algorithms have been proposed. Long et al. proposed a distributed multi-robot navigation algorithm that directly maps sensor data to robot behavior actions. This algorithm accelerates the training efficiency of the parallel PPO algorithm through a multi-scene, multi-stage learning framework. However, the obstacle avoidance strategy trained by this algorithm still cannot consistently generate perfect behavior actions in complex scenarios. On this basis, Fan et al. combined deep reinforcement learning strategies with traditional control strategies and designed multiple sub-strategies according to different scenarios. The robot judges the current scenario based on the sensors and then selects different sub-strategies. The algorithm is easy to port and robust in simple scenarios. However, it relies on human-designed strategy selection rules. In real-world scenarios, the sensors on the robot have errors, and the scenarios encountered by the robot are more complex. This algorithm can easily lead to decision-making errors by the robot.

[0005] Therefore, in existing solutions that combine deep reinforcement learning strategies with traditional control strategies, the over-reliance on manually designed strategy selection rules results in poor robot recognition of the current environment, inability to adapt to complex real-world scenarios, and a lack of real-time performance and safety in obstacle avoidance and navigation. Summary of the Invention

[0006] In view of this, the present invention provides a multi-robot safe navigation method and system based on hierarchical deep reinforcement learning. By designing multiple sub-behavioral strategies based on traditional control rules and introducing behavior selection strategies based on deep reinforcement learning, the method eliminates the dependence on manually designed strategy selection rules, enhances the robot's ability to recognize the current environment, and improves the real-time performance and safety of multi-robot obstacle avoidance navigation.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0008] Step 1: For the multi-robot navigation problem model, construct a hierarchical deep reinforcement learning network framework; the network framework includes six modules: input module, feature extraction module, behavior selection module, DRL control module, auxiliary control module, and output module.

[0009] The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module, and the target point coordinates are sent to the auxiliary control module.

[0010] The feature extraction module consists of a one-dimensional convolutional network, which is used to extract environmental features from the LiDAR search frame. The extracted environmental features are then fed into the behavior selection module and the DRL control module.

[0011] The behavior selection module consists of a fully connected layer. Based on environmental features, target point coordinates, and robot speed, it generates and outputs discrete action values. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy.

[0012] The DRL control module consists of a fully connected layer, which is used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates, and velocity, and generates robot obstacle avoidance control commands.

[0013] The auxiliary control module is used to execute the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates and generates target-driven control commands. The safe steering strategy controls the robot to brake and turn in an emergency and generates steering control commands.

[0014] The output module is used to determine and output the control command of a sub-behavior strategy based on the output of the behavior selection module, so as to control the robot.

[0015] Step 2: Train the DRL control module.

[0016] Step 3: Train the behavior selection module.

[0017] Step 4: Use the hierarchical deep reinforcement learning network framework trained in Steps 2 and 3 to perform navigation control on the multi-robot system.

[0018] Furthermore, the multi-robot navigation problem model is as follows:

[0019] The multi-robot navigation problem is modeled as a partially observable Markov decision process (POMDP). The POMDP consists of a six-tuple (S, A, R, P, Ω, O), where S is the state space, A is the action function, R is the reward function, P is the state transition model, Ω is the observation space, and O is the observation probability distribution of the system state.

[0020] At each time step t, the i-th robot observes According to the sharing policy Execute action Receive rewards And according to the state transition model Enter the next state

[0021] The goal of POMDP is to find the optimal shared strategy for all robots by minimizing the time it takes for all robots to reach the target in the same scenario, expressed as: in It is the navigation time for the i-th robot to reach the target.

[0022] Furthermore, the input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts, specifically:

[0023]

[0024] Among them o t This is for the robot's observation data of the environment; These are the last three consecutive frames of LiDAR data for the robot at the current moment. The coordinates of the target point in the robot coordinate system. The current velocity of the robot is represented by linear velocity and angular velocity, where the linear velocity ranges from [0,1] and the angular velocity ranges from [-1,1].

[0025] Furthermore, the robot obstacle avoidance control commands generated by the DRL control module include the linear velocity and angular velocity that the robot should take. Two different activation functions are set in the output layer of the DRL control module. One is the sigmoid function, which constrains the output of the linear velocity control command to [0,1], and the other is the tanh function, which constrains the output of the angular velocity control command to [-1,1].

[0026] Furthermore, in the auxiliary control module, the control rules for the safe steering strategy are as follows: the linear velocity is set to 0, and the angular velocity is taken from the angular velocity generated by the DRL strategy.

[0027] Furthermore, in the auxiliary control module, the target-driven control commands generated by the target-driven strategy include the linear velocity and angular velocity that the robot should take, determined by a pre-set adjustable parameter p(p l ,p r Control is implemented, and the control rules are as follows:

[0028]

[0029] Where vt w is the linear velocity at time t. t p is the angular velocity at time t; l ,p r These are preset adjustable parameters for linear velocity and angular velocity, respectively. Let be the robot's coordinates in the map coordinate system at time t; Let t be the robot's orientation angle in the map coordinate system at time t; The coordinates of the target point in the map coordinate system at time t.

[0030] Further, step 2: Training the DRL control module, specifically:

[0031] The DRL control module is trained using the parallel TD3 algorithm, and the reward / penalty function is defined as follows:

[0032] R l =R lg +R lc +R lv

[0033]

[0034]

[0035] R lv =γ v (v-|w|)

[0036] Where R l Overall rewards and punishments, R lg Reward for reaching the goal, R lc Collision penalty, R lv Speed ​​bonus, r reach The reward coefficient for reaching the goal, γ g The reward coefficient, r, for approaching the goal crash Collision penalty coefficient, γ v The speed bonus coefficient, v is the robot's linear velocity, w is the robot's angular velocity, r0 represents the minimum value of the LiDAR in the current frame, r t Let r be the distance between the robot and the target point at the current time t. t-1 Let r be the distance between the robot and the target point at the previous moment. c This represents the minimum distance for a robot to collide.

[0037] Further, step 3: training the behavior selection module, specifically:

[0038] The discrete action value output by the behavior selection module is used by the robot to select different sub-behavior strategies. This discrete action value is a. h ∈{0,1,2}, if a h =0, the robot chooses the DRL strategy; if ah =1, the robot selects a target-driven strategy; if a h =2, the robot selects a safe turning strategy;

[0039] The behavior selection module is trained using the parallel D3QN algorithm, and the reward / penalty function is defined as follows:

[0040] R h =R hg +R hc +R ha

[0041]

[0042]

[0043]

[0044] Where R hg Represents the target distance reward, encouraging the robot to move towards the target point and used to generate continuous rewards; R hc Indicates collision penalty; R ha Indicates action reward, r reach r represents the reward coefficient for reaching the goal. freeze r represents the penalty coefficient for freezing the robot. crash Collision penalty coefficient, γ g The reward coefficient for approaching the target, r0 represents the minimum value of the lidar in the current frame, r t Let r be the distance between the robot and the target point at the current time t. t-1 Let r be the distance between the robot and the target point at the previous moment. c This represents the minimum distance for a robot to collide.

[0045] Another embodiment of the present invention provides a multi-robot safety navigation system based on hierarchical deep reinforcement learning, comprising six modules: an input module, a feature extraction module, a behavior selection module, a DRL control module, an auxiliary control module, and an output module.

[0046] The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module, and the target point coordinates are sent to the auxiliary control module.

[0047] The feature extraction module consists of a one-dimensional convolutional network, which is used to extract environmental features from the LiDAR search frame. The extracted environmental features are then fed into the behavior selection module and the DRL control module.

[0048] The behavior selection module consists of a fully connected layer. Based on environmental features, target point coordinates, and robot speed, it generates and outputs discrete action values. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy.

[0049] The DRL control module consists of a fully connected layer, which is used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates, and velocity, and generates robot obstacle avoidance control commands.

[0050] The auxiliary control module is used to execute the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates and generates target-driven control commands. The safe steering strategy controls the robot to brake and turn in an emergency and generates steering control commands.

[0051] The output module is used to determine and output the control command of a sub-behavior strategy based on the output of the behavior selection module, so as to control the robot.

[0052] Beneficial effects:

[0053] 1. This invention provides a multi-robot safe navigation method based on hierarchical deep reinforcement learning. This scheme employs a skill-based hierarchical deep reinforcement learning model framework. By designing rule-based target-driven strategies and safe steering strategies, as well as obstacle avoidance navigation strategies based on deep reinforcement learning, it implements three low-level behavioral strategies: target approach, emergency braking, and obstacle avoidance navigation. High-level behavioral selection strategies are trained through deep reinforcement learning. A well-learned behavioral selection model effectively avoids reliance on manually designed strategy selection rules. This invention enhances the robot's ability to recognize the current environment and improves the speed and safety of multi-robot obstacle avoidance navigation by designing multiple sub-behavioral strategies based on traditional control rules and introducing behavior selection strategies based on deep reinforcement learning.

[0054] 2: The multi-robot safe navigation method based on hierarchical deep reinforcement learning provided by this invention can effectively improve the speed and safety of multi-robot navigation by introducing a target-driven strategy and a safe turning strategy.

[0055] 3: The multi-robot safe navigation method based on hierarchical deep reinforcement learning provided by this invention avoids the reliance on manually designed and selected strategies by adopting a skill-based hierarchical deep reinforcement learning framework, and can effectively improve the navigation performance of robots in complex environments including corridors, blind spots and other structures.

[0056] 4. The multi-robot safety navigation system based on hierarchical deep reinforcement learning provided by this invention is an obstacle avoidance navigation model implemented under non-communication conditions of multiple robots, based on the above-mentioned multi-robot safety navigation method. The trained multi-robot safety navigation system is easy to port and is applicable not only to multiple robots but also to single robots, and can also be extended to large robot systems. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the network model framework based on hierarchical deep reinforcement learning provided by the present invention;

[0058] Figure 2 This is a schematic diagram of the training process simulation environment provided by the present invention; Figure 2 (a), (b), (c), and (d) in the figure are schematic diagrams of the simulation environment for the training process of the circular scene, the corridor scene, the random scene, and the maze scene, respectively.

[0059] Figure 3 This is a schematic diagram of the testing process simulation environment provided by the present invention; Figure 3 (a) and (b) in the diagram are schematic diagrams of the simulation environment for the test process in a random scenario and a maze scenario, respectively.

[0060] Figure 4 This is a schematic diagram of the visual navigation path for the testing process provided by the present invention; Figure 4 (a), (b), (c), and (d) in the diagram are visualization navigation paths for the testing process of the four methods: SLCAP, Hybrid-RL, RL (TD3), and HDRL, respectively. Detailed Implementation

[0061] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0062] This invention provides a multi-robot safe navigation method based on hierarchical deep reinforcement learning. It adopts a skill-based hierarchical deep reinforcement learning model framework, designs multiple sub-behavioral strategies based on traditional control rules, and introduces a behavior selection strategy based on deep reinforcement learning. This enhances the robot's ability to recognize the current environment and effectively improves the speed and safety of multi-robot obstacle avoidance navigation.

[0063] Step 1: Construct a hierarchical deep reinforcement learning network framework for the multi-robot navigation problem model.

[0064] The multi-robot navigation problem model is as follows:

[0065] The multi-robot navigation problem is modeled as a partially observable Markov decision process (POMDP). The POMDP consists of a six-tuple (S, A, R, P, Ω, O), where S is the state space, A is the action function, R is the reward function, P is the state transition model, Ω is the observation space, and O is the observation probability distribution of the system state.

[0066] At each time step t, the i-th robot observes According to the sharing policy Execute action Receive rewards And according to the state transition model Enter the next state

[0067] The goal of POMDP is to find the optimal shared strategy for all robots by minimizing the time it takes for all robots to reach the target in the same scenario, expressed as: in It is the navigation time for the i-th robot to reach the target.

[0068] Figure 1 The network framework proposed in this invention comprises six modules: an input module, a feature extraction module, an action selection module, a DRL control module, an auxiliary control module, and an output module.

[0069] The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module, and the target point coordinates are sent to the auxiliary control module.

[0070] Input module: Each robot's observation of the current environment consists of three parts: in These are the last three consecutive frames of LiDAR data for the robot at the current moment. The coordinates of the target point in the robot coordinate system. Let be the robot's linear velocity and angular velocity at the current moment. The former has a value range of [0,1], and the latter has a value range of [-1,1].

[0071] The feature extraction module, composed of a one-dimensional convolutional network, is used to extract environmental features from the LiDAR search frames. The extracted environmental features are then fed into the behavior selection module and the DRL control module. To alleviate the problem of the robot's state not being uniquely determined due to partially observable data, this invention selects the last three consecutive frames of LiDAR data at the current moment as input. Due to the large amount of data, the feature extraction module, composed of a one-dimensional convolutional network, is used first to extract environmental features.

[0072] The behavior selection module consists of a fully connected layer. Based on environmental features, target point coordinates, and robot speed, it generates and outputs discrete action values. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy.

[0073] The DRL control module consists of a fully connected layer used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates, and velocity, generating obstacle avoidance control commands. The obstacle avoidance control commands generated by the DRL control module include the linear velocity and angular velocity that the robot should take. Two different activation functions are set in the output layer of the DRL control module: one is the sigmoid function, which constrains the output of the linear velocity control command to [0,1], and the other is the tanh function, which constrains the output of the angular velocity control command to [-1,1].

[0074] The auxiliary control module executes the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates, generating target-driven control commands. The target-driven control commands generated by the target-driven strategy include the linear velocity and angular velocity that the robot should adopt, determined by a pre-set adjustable parameter p(p...). l ,p r Control is implemented, and the control rules are as follows:

[0075]

[0076] Where v t w is the linear velocity at time t. t p is the angular velocity at time t; l ,p r These are preset adjustable parameters for linear velocity and angular velocity, respectively. Let be the robot's coordinates in the map coordinate system at time t; Let t be the robot's orientation angle in the map coordinate system at time t; The coordinates of the target point in the map coordinate system at time t.

[0077] The safe steering strategy is used to control the robot to brake and turn in an emergency, generating steering control commands. The safe steering strategy aims to enable the robot to brake and turn in emergency situations, enhancing the safety of robot navigation. The control rules of the safe steering strategy are: set the linear velocity to 0, and set the angular velocity to the angular velocity generated by the DRL strategy.

[0078] The output module determines and outputs a control command for a sub-behavior strategy based on the output of the behavior selection module, for controlling the robot. In this embodiment, the output module outputs the linear velocity and angular velocity control command 'a' that the robot should take. t =[v t ,w t ], all are continuous values, where v t ∈(0,1),w t ∈(-1,1). It should be noted that if there is a significant difference between the robot's current speed state and the output speed control command, the robot needs to gradually reach the speed value corresponding to the control command through acceleration, rather than reaching it instantaneously.

[0079] Step 2: Train the DRL control module.

[0080] Based on the completed model construction, the DRL control module is first trained. A deep reinforcement learning-based obstacle avoidance navigation strategy is learned using the parallel TD3 algorithm, and the reward / penalty function is defined as follows:

[0081] R l =R lg +R lc +R lv

[0082]

[0083]

[0084] R lv =γ v (v-|w|)

[0085] Where r0 is the minimum value of the lidar in the current frame, r t Let r be the distance between the robot and the target point at the current moment. t-1 Let r be the distance between the robot and the target point at the previous moment. c R is the minimum distance for robot collisions. l For the overall rewards and punishments, R lg The target distance reward is used to encourage the robot to move towards the target point; R lc Collision penalty is a larger penalty imposed when the robot collides with another robot; R lvAs a speed reward, the robot is encouraged to move towards the target point quickly along a smooth trajectory; reach γ is the reward coefficient for reaching the goal. g The reward coefficient for approaching the goal, r crash This represents the collision penalty coefficient. In this embodiment of the invention, the parameters set during training are as follows: r reach =15,γ g =2.5,r crash =-15 and γ v =0.1.

[0086] The training steps for the DRL control module are as follows:

[0087] (1) Four 20*20 simulation environments were created using the ROS Stage simulator. Each environment contained eight isomorphic robots and several obstacles. The starting point of each robot was fixed, and the target point was randomly generated in each round of training. The simulation round ended when the mobile robot collided or reached the target point.

[0088] (2) Initialize network parameters and experience pool.

[0089] (3) Each robot generates action a according to the shared policy network. t It continuously interacts with the simulation environment and collects quadruplets (o t ,a t ,r t ,o t+1 The data is collected and stored in the shared experience pool.

[0090] (4) Update the policy network and evaluation network using historical data from the shared experience pool, and repeat this process until the model converges to obtain a DRL policy with good obstacle avoidance and navigation performance.

[0091] Step 3: After training the DRL control module, the behavior selection module is further trained. The output of this module is discrete and used by the robot to select different sub-behavior strategies; it is defined as a. h ∈{0,1,2}, if a h =0, the robot chooses the DRL strategy; if a h =1, the robot selects a target-driven strategy; if a h =2, the robot selects a safe turning strategy. The behavior selection module is trained using the parallel D3QN algorithm, and the reward / penalty function is defined as:

[0092] R h =R hg +R hc +R ha

[0093]

[0094]

[0095]

[0096] Where R h For the total reward; R hg Represents the target distance reward, encouraging the robot to move towards the target point. It is used to generate continuous rewards and avoid overly sparse rewards during training; R hc This represents a collision penalty, inflicted with a larger penalty when the robot collides; R ha This indicates a reward for actions, used to prevent the robot from frequently choosing safe turning strategies, which could lead to the robot freezing or spinning in place; r reach r is the reward coefficient for reaching the goal. freeze r is the penalty coefficient for freezing the robot. crash r is the collision penalty coefficient, set during training. reach =15,γ g =2.5,r crash =-15 and r freeze =-0.5; γ g The reward coefficient for approaching the target is r0, which is the minimum value of the lidar in the current frame. t Let r be the distance between the robot and the target point at the current time t. t-1 Let r be the distance between the robot and the target point at the previous moment. c This represents the minimum distance for a robot to collide.

[0097] The training steps for the behavior selection module are as follows:

[0098] (1) Set the trained DRL strategy, target-driven strategy and safe steering strategy as the robot's sub-behavior strategies and interact with the simulation environment.

[0099] (2) Initialize network parameters and experience pool.

[0100] (3) Each robot selects a sub-behavior strategy based on the output of the behavior selection module, and executes the action n times based on the output of the sub-behavior strategy, collecting the quadruple (o t ,a h ,r t ,o t+1 The data is collected and stored in the shared experience pool.

[0101] (4) Update the action value network using historical data in the shared experience pool, and repeat this process until the model converges to obtain a behavior selection model with good obstacle avoidance and navigation performance.

[0102] The method of this invention trains multiple robots in the same simulation environment and shares the policy network. This not only allows for the collection of more historical data, making the policy network training faster, but also enhances the generalization ability of the policy network through the constantly changing environment.

[0103] Step 4: Use the hierarchical deep reinforcement learning network framework trained in Steps 2 and 3 to perform navigation control on the multi-robot system.

[0104] exist Figure 2 The DRL control module and behavior selection module are trained in the simulation environment shown. Figure 2 This is a schematic diagram of the training process simulation environment provided by the present invention; Figure 2 (a), (b), (c), and (d) are schematic diagrams of the simulation environment for the training process of a circular scene, a corridor scene, a random scene, and a maze scene, respectively. The trained multi-robot obstacle avoidance navigation framework is compared with existing methods, and 100 evaluations are performed for each method. The evaluation results are shown in Table 1.

[0105] Table 1

[0106]

[0107] It can be seen that Hybrid-RL achieved the best navigation results in circular scenarios, because circular scenarios are the simplest, containing only other robots and no obstacles. In other more complex scenarios, the method of this invention has higher success rates and lower collision rates, and also has overall advantages in other parameter indicators. This shows that the method of this invention is effective for multi-robot navigation tasks in complex scenarios.

[0108] To test the generalization ability of the method of this invention, the trained multi-robot obstacle avoidance navigation framework was deployed to, for example, Figure 3 In the two completely new and complex environments shown, Figure 3 This is a schematic diagram of the testing process simulation environment provided by the present invention; Figure 3 (a) and (b) in the table are schematic diagrams of the simulation environment of the test process for the random scenario and the maze scenario, respectively. The two scenarios were evaluated 100 times each, and the evaluation results are shown in Table 2.

[0109] Table 2

[0110]

[0111] The method of this invention still has the highest success rate and the lowest collision rate, as shown in the visualization results. Figure 4 As shown, Figure 4 This is a schematic diagram of the visual navigation path for the testing process provided by the present invention; Figure 4(a), (b), (c), and (d) in the figure are visualization navigation path diagrams of the test process of the four methods SLCAP, Hybrid-RL, RL (TD3), and HDRL, respectively. It can be seen that the trajectory generated by the method of the present invention is smoother. For example, when the robot approaches the target point, the robot will drive directly to the target point instead of lingering at the target point like other methods.

[0112] Therefore, this invention provides a multi-robot safety navigation system based on hierarchical deep reinforcement learning, comprising six modules: an input module, a feature extraction module, a behavior selection module, a DRL control module, an auxiliary control module, and an output module.

[0113] The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module, and the target point coordinates are sent to the auxiliary control module.

[0114] The feature extraction module consists of a one-dimensional convolutional network, which is used to extract environmental features from the LiDAR search frame. The extracted environmental features are then fed into the behavior selection module and the DRL control module.

[0115] The behavior selection module consists of a fully connected layer. Based on environmental features, target point coordinates, and robot speed, it generates and outputs discrete action values. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy.

[0116] The DRL control module consists of a fully connected layer, which is used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates, and velocity, and generates robot obstacle avoidance control commands.

[0117] The auxiliary control module is used to execute the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates and generates target-driven control commands. The safe steering strategy controls the robot to brake and turn in an emergency and generates steering control commands.

[0118] The output module is used to determine and output the control command of a sub-behavior strategy based on the output of the behavior selection module, so as to control the robot.

[0119] This system enhances the robot's ability to recognize the current environment by designing multiple sub-behavioral strategies based on traditional control rules and introducing behavior selection strategies based on deep reinforcement learning. This effectively improves the speed and safety of multi-robot obstacle avoidance navigation.

[0120] This system is an obstacle avoidance and navigation model implemented under non-communication conditions for multiple robots. Once trained, the system is easy to port and is applicable not only to multiple robots but also to single robots, and can also be extended to large robot systems.

[0121] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-robot safe navigation method based on hierarchical deep reinforcement learning, characterized in that, The steps of this method include: Step 1: For the multi-robot navigation problem model, construct a hierarchical deep reinforcement learning network framework; the network framework includes six modules: input module, feature extraction module, behavior selection module, DRL control module, auxiliary control module, and output module; The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module. The target point coordinates are sent to the auxiliary control module. The feature extraction module consists of a one-dimensional convolutional network, which is used to extract environmental features from the LiDAR search frame. The extracted environmental features are then sent to the behavior selection module and the DRL control module. The behavior selection module consists of a fully connected layer. It generates and outputs discrete action values ​​based on environmental features, target point coordinates, and robot speed. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy. The DRL control module consists of a fully connected layer and is used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates and velocity, and generates robot obstacle avoidance control commands. The auxiliary control module is used to execute the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates and generates target-driven control commands. The safe steering strategy controls the robot to brake and turn in an emergency and generates steering control commands. The output module is used to determine and output a control command for a sub-behavior strategy based on the output of the behavior selection module, for controlling the robot. Step 2: Train the DRL control module; Step 3: Train the behavior selection module; Step 4: Use the hierarchical deep reinforcement learning network framework trained in Steps 2 and 3 to perform navigation control on the multi-robot system.

2. The multi-robot safe navigation method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that, The multi-robot navigation problem model is as follows: The multi-robot navigation problem is modeled as a partially observable Markov decision process (POMDP). The POMDP consists of a six-tuple (S, A, R, P, Ω, O), where S is the state space, A is the action function, R is the reward function, P is the state transition model, Ω is the observation space, and O is the observation probability distribution of the system state. At each time step t, the i-th robot observes According to the sharing policy Execute action Receive rewards And according to the state transition model Enter the next state The goal of POMDP is to find the optimal shared strategy for all robots by minimizing the time it takes for all robots to reach the target in the same scenario, expressed as: in It is the navigation time for the i-th robot to reach the target.

3. The multi-robot safe navigation method based on hierarchical deep reinforcement learning according to claim 1, characterized in that: The input module is used to receive and acquire the observation data of each robot on the environment, and includes three parts, specifically: Among them o t This is for the robot's observation data of the environment; These are the last three consecutive frames of LiDAR data for the robot at the current moment. The coordinates of the target point in the robot coordinate system. The current velocity of the robot is represented by linear velocity and angular velocity, where the linear velocity ranges from [0,1] and the angular velocity ranges from [-1,1].

4. A multi-robot safe navigation method based on hierarchical deep reinforcement learning according to claim 3, characterized in that: The robot obstacle avoidance control command generated by the DRL control module includes the linear velocity and angular velocity that the robot should take. Two different activation functions are set in the output layer of the DRL control module. One is the sigmoid function, which constrains the output of the linear velocity control command to [0,1], and the other is the tanh function, which constrains the output of the angular velocity control command to [-1,1].

5. A multi-robot safe navigation method based on hierarchical deep reinforcement learning according to claim 4, characterized in that: In the auxiliary control module, the control rule for the safe steering strategy is: the linear velocity is set to 0, and the angular velocity is taken from the angular velocity generated by the DRL strategy.

6. A multi-robot safe navigation method based on hierarchical deep reinforcement learning according to claim 4, characterized in that: In the auxiliary control module, the target drive strategy generates target drive control commands including the linear velocity and angular velocity that the robot should take, determined by a pre-set adjustable parameter p(p l ,p r Control is implemented, and the control rules are as follows: Where v t w is the linear velocity at time t. t p is the angular velocity at time t; l ,p r These are preset adjustable parameters for linear velocity and angular velocity, respectively. Let be the robot's coordinates in the map coordinate system at time t; Let t be the robot's orientation angle in the map coordinate system at time t; The coordinates of the target point in the map coordinate system at time t.

7. The multi-robot safe navigation method based on hierarchical deep reinforcement learning according to any one of claims 1 to 6, characterized in that, Step 2: Training the DRL control module specifically involves: The DRL control module is trained using the parallel TD3 algorithm, and the reward / penalty function is defined as follows: R l =R lg +R lc +R lv R lv =γ v (v-|w|) Where R l For the overall rewards and punishments, R lg As a reward for reaching the goal, R lc For collision penalty, R lv For speed reward, r reach γ is the reward coefficient for reaching the goal. g The reward coefficient for approaching the goal, r crash γ is the collision penalty coefficient. v Here, v is the robot's linear velocity, w is the robot's angular velocity, r0 is the minimum value of the LiDAR in the current frame, and r... t Let r be the distance between the robot and the target point at the current time t. t-1 Let r be the distance between the robot and the target point at the previous moment. c This represents the minimum distance for a robot to collide.

8. A multi-robot safe navigation method based on hierarchical deep reinforcement learning according to claim 1, 3, or 5, characterized in that: Step 3: Training the behavior selection module specifically involves: The discrete action value output by the behavior selection module is used by the robot to select different sub-behavior strategies. This discrete action value is a. h ∈{0,1,2}, if a h =0, the robot chooses the DRL strategy; if a h =1, the robot selects a target-driven strategy; if a h =2, the robot selects a safe turning strategy; The behavior selection module is trained using the parallel D3QN algorithm, and the reward / penalty function is defined as follows: R h =R hg +R hc +R ha Where R h For the total reward; R hg The target distance reward incentivizes the robot to move towards the target point, thus generating continuous rewards; R hc For collision penalty; R ha For action rewards, r reach r is the reward coefficient for reaching the goal. freeze r is the penalty coefficient for freezing the robot. crash γ is the collision penalty coefficient. g The reward coefficient for approaching the target is r0, which is the minimum value of the lidar in the current frame. t Let r be the distance between the robot and the target point at the current time t. t-1 Let r be the distance between the robot and the target point at the previous moment. c This represents the minimum distance for a robot to collide.

9. A multi-robot safety navigation system based on hierarchical deep reinforcement learning, characterized in that, It includes six modules: input module, feature extraction module, behavior selection module, DRL control module, auxiliary control module, and output module; The input module is used to receive and acquire the observation data of each robot on the environment, which includes three parts: the robot's real-time LiDAR search frame, the target point coordinates in the robot coordinate system, and the real-time robot speed. The LiDAR search frame is sent to the feature extraction module, and the combination of the target point coordinates and the robot speed is sent to the behavior selection module. The output of the feature extraction module, the combination of the target point coordinates and the robot speed is sent to the DRL control module. The target point coordinates are sent to the auxiliary control module. The feature extraction module consists of a one-dimensional convolutional network, which is used to extract environmental features from the LiDAR search frame. The extracted environmental features are then sent to the behavior selection module and the DRL control module. The behavior selection module consists of a fully connected layer. It generates and outputs discrete action values ​​based on environmental features, target point coordinates, and robot speed. The discrete action values ​​correspond to three sub-behavior strategies, including DRL strategy, target-driven strategy, and safe steering strategy. The DRL control module consists of a fully connected layer and is used to execute the DRL strategy. The DRL strategy controls the robot to avoid obstacles based on a combination of environmental features, target point coordinates and velocity, and generates robot obstacle avoidance control commands. The auxiliary control module is used to execute the target-driven strategy and the safe steering strategy. The target-driven strategy drives the robot to move towards the target based on the target point coordinates and generates target-driven control commands. The safe steering strategy controls the robot to brake and turn in an emergency and generates steering control commands. The output module is used to determine and output a control command for a sub-behavior strategy based on the output of the behavior selection module, for controlling the robot.

Images