Mobile robot autonomous obstacle avoidance method based on multi-thread asynchronous deep reinforcement learning

Abstract

The invention discloses a mobile robot autonomous obstacle avoidance method based on multi-thread asynchronous deep reinforcement learning, and the method specifically comprises the steps: adding a multi-task asynchronous parallel mechanism on the basis of a PPO algorithm, constructing MAPPO, separating different obstacle avoidance task scenes, and training the scenes at the same time; the method comprises the following steps: on the basis of nokov-lidar multi-sensor sensing information, constructing a robot environment state observation space; designing a discretized action space based on the global grid world navigation map, and setting kinematics constraints for state updating; designing a navigation reward function, and guiding the mobile robot to make an optimal obstacle avoidance decision in a complex environment; an early collision prediction module is established based on a multi-layer perception mechanism, collision information from a perceptible environment is deduced, and an optimal obstacle avoidance strategy is trained in combination with MAPPO learning. According to the invention, sufficient mobile robot-environment interaction can be realized, the exploration capability of the robot action decision model is improved, and real-time obstacle avoidance of the robot in the process of moving to the target is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep reinforcement learning decision-making and robot path planning, and specifically relates to an autonomous obstacle avoidance method for a mobile robot based on multi-threaded asynchronous deep reinforcement learning. Background Art

[0002] Typically, in practical applications involving complex dynamic scenarios, the safety and reliability of mobile robots are considered the two most important criteria that free-ranging robotic vehicles must meet before deployment and production. When a robotic vehicle encounters a moving obstacle while moving within its field of view, it must take proactive action to avoid the obstacle. Because the motion state of moving obstacles is uncertain, accurately extracting motion features and effectively predicting the trajectory of moving obstacles in complex dynamic environments is often very difficult, making it difficult for mobile robots to perform correct navigation actions to avoid collisions. Therefore, developing safe, efficient, and reliable obstacle avoidance algorithms in complex dynamic scenarios is a key scientific and technological issue in the mobile robot navigation phase.

[0003] The problem of dynamic obstacle avoidance for mobile robots has been the subject of extensive research in the field of robotics and other related fields. Based on the task hierarchy, current collision-free navigation methods can be roughly divided into map-based global obstacle avoidance navigation and perception-based local obstacle avoidance navigation. The former achieves collision avoidance by generating a continuous path based on a known obstacle map. This process involves two key points: first, offline / online construction of an environment map with obstacles using SLAM technology, and then planning collision-free trajectories based on the map to achieve safe navigation. However, in complex dynamic scenes filled with moving obstacles, SLAM may fail due to noise or occlusion, resulting in the inability to generate a reliable map.

[0004] Given these limitations, some recent research has proposed local collision avoidance navigation techniques, a solution for mapless navigation. Traditional mapless obstacle avoidance navigation methods utilize geometric rule-based strategies to simplify obstacles into convex bodies and then calculate the upper and lower bounds of a collision-free navigation path. Specifically, these methods exploit the repulsive forces generated when a robot approaches an obstacle to drive its avoidance. While efficient and easy to implement, these solutions significantly degrade when the dynamics of moving obstacles vary significantly. With the advancement of artificial intelligence, a growing number of researchers have focused on deep reinforcement learning techniques to address the limitations of these traditional mapless collision avoidance processes. Deep reinforcement learning is a machine learning algorithm that learns through trial and error and predicts navigation actions directly from the robot's sensor perception. These learning-based mapless obstacle avoidance navigation techniques learn and optimize policies through extensive interaction with the environment to achieve specific obstacle avoidance tasks and maximize cumulative reward. While these solutions perform well in relatively simple and structured test environments, these learning-based obstacle avoidance models struggle to generalize to more complex and highly dynamic scenarios. Summary of the Invention

[0005] In response to the above-mentioned shortcomings of the existing technology, the present invention proposes an autonomous obstacle avoidance method for mobile robots based on multi-threaded asynchronous deep reinforcement learning to achieve full mobile robot-environment interaction, maximize the exploration level of the robot action decision model in complex dynamic scenes, and ensure that the mobile robot successfully avoids collisions while moving towards the target.

[0006] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0007] The autonomous obstacle avoidance method for mobile robots based on multi-threaded asynchronous deep reinforcement learning specifically includes the following steps:

[0008] S1. Based on the PPO algorithm, a multi-task asynchronous parallel mechanism is added to build a multi-threaded asynchronous deep reinforcement learning architecture MAPPO, which separates different obstacle avoidance task scenarios and trains them simultaneously;

[0009] S2. Based on the nokov-lidar multi-sensor perception information, a robot environment state observation space is constructed to provide environment understanding and state estimation for the mobile robot;

[0010] S3. Based on Nokov's global grid world navigation map, design a discretized action space and set kinematic constraints for the mobile robot's state update during the action decision process;

[0011] S4. Design a navigation reward function to guide the mobile robot to make the best obstacle avoidance decisions in complex environments;

[0012] S5. Build a premature collision prediction module PCP based on a multi-layer perceptron to infer collision information from the perceptible environment, and combine it with the multi-threaded asynchronous deep reinforcement learning architecture MAPPO to learn and train the optimal obstacle avoidance strategy.

[0013] Furthermore, step S1 specifically includes:

[0014] S11. Use the Trust Region Policy Optimization algorithm TRPO to describe the strategy optimization problem of mobile robot path planning. The formula is expressed as:

[0015]

[0016] in, and π θ They are the old strategy before the strategy network is updated and the new strategy after the update; is the advantage function, calculated based on the generalized advantage estimation algorithm; X t 、a t Represent the state and action at time t respectively Express expectations, Represents the distribution of state-action trajectories;

[0017] S12. Construct a loss function based on the PPO-Clip algorithm to simplify the optimization problem described by the TRPO algorithm; the constructed loss function The formula is expressed as:

[0018]

[0019] in, Represents the current policy π θ The action probability under the previous strategy The ratio of the action probabilities under t When (θ)∈[1-∈,1+∈], the function outputs Otherwise, output the upper bound 1+∈ or the lower bound 1-∈;

[0020] S13. Minimize the mean square error based on the gradient descent algorithm to update the value network of PPO-Clip; the formula is expressed as:

[0021]

[0022] Among them, ω is the value network update parameter; r t′ is the reward at time t′; V represents the reward that the robot will get starting from t′; ω (X t ) represents the state value; γ is a hyperparameter used to adjust the variance;

[0023] S14. Based on the PPO-clip strategy calculation, a multi-task asynchronous parallel mechanism is added to build the MAPPO architecture. In the MAPPO architecture, each Worker child node process randomly assigns a task scenario to the mobile robot for interaction and data collection. At the same time, the gradient value is calculated based on the interaction trajectory and sent to the Global master node.

[0024] Furthermore, step S2 is specifically as follows:

[0025] Construct the robot environment state observation space, and the state observation sequence of each time step is recorded as:

[0026]

[0027] Among them, l t It is the distance point cloud data provided by the laser radar scanning one week; t A 2D vector representing the rotational and translational velocities of the mobile robot at time t; and It is a 2D navigation path constraint constructed based on the global guidance path and the robot's own state under Nokov global perception; t is the one-dimensional heading angle;

[0028] Superimpose three consecutive historical observation sequences as the current state input for each time step, that is:

[0029] X t =(x t-2 , x t-1 , x t );

[0030] Among them, X t Represents the state input at the current time t, x t-2 , x t-1 , x t are the state observation sequences at time t-2, t-1, and t respectively.

[0031] Furthermore, the action space designed in step S3 is specifically:

[0032] The discretized motion of the designed robot is: in,

[0033]

[0034] v l and v w It is the reference value of translation speed and rotation speed set based on the environment;

[0035] Add the discretized action to the action space S, the formula is expressed as:

[0036]

[0037] The action space contains 28 actions including stop actions.

[0038] Furthermore, in step S3, during the action decision process, action constraints are set for the robot state update, that is, the formula for the robot state update within the time interval Δt is expressed as:

[0039]

[0040] in, and are the robot's position p at time t t The horizontal and vertical components of t is the one-dimensional heading angle; and are translational velocity and rotational velocity respectively; is the state vector at time t′ after the action is executed, Δt=t′-t.

[0041] Furthermore, step S4 is specifically as follows:

[0042] Consider moving the robot at maximum speed And with The same direction, ideally traveled a desired distance The navigation reward function at each moment during this period is calculated as:

[0043]

[0044] in, is the time step penalty, which is used to encourage the robot to shorten the decision time; is the tangential motion reward function, which is used to constrain the robot to move along the direction of the global guidance path; is the normal motion reward function; Penalty for collision; To reach the reward;

[0045] The formula is:

[0046]

[0047] in, Indicates the movement distance of the robot in the tangent direction of the global guidance path.

[0048] The formula is:

[0049]

[0050] in, Represents the distance traveled by the robot in the normal direction of the global guidance vector; and They are and exist Normal vector The module of the projection;

[0051] The formula is expressed as:

[0052]

[0053] The formula is expressed as:

[0054]

[0055] Among them, B k (0≤k≤M) represents the position of the kth obstacle; p g is the target point position; r collision and r goal There are two types of sparse feedback used to adjust the robot's behavior to meet physical hard constraints.

[0056] Furthermore, step S5 specifically includes:

[0057] S51, based on the multi-layer perceptron design premature collision module PCP, the state of each time step X t As the current input of PCP, calculate all possible actions ca at each time step t The collision probability PCP(ca t |X t ), each possible action is labeled as:

[0058] ca t ~PCP(ca t |X t );

[0059] S52. Use multi-label classification loss as the loss function for PCP module training The formula is as follows:

[0060]

[0061] in, is the distance from the robot's current position to its local track point g b The preferred action of ,the local track points are obtained by discretizing the global guidance path; yes The delta function at , which is supervised by the global guidance path;

[0062] S53. Introduce the premature collision prediction module into MAPPO to jointly learn the navigation strategy; that is, construct a joint learning loss function As the total loss function of MAPPO after the introduction of the PCP module, is a loss function constructed based on the PPO-Clip algorithm; the total loss function guides the model to learn the navigation strategy and obtain the optimal obstacle avoidance strategy.

[0063] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0064] The present invention proposes an autonomous obstacle avoidance method for mobile robots based on multi-threaded asynchronous deep reinforcement learning. First, a multi-task asynchronous parallel mechanism is introduced into the PPO algorithm to construct a multi-threaded asynchronous deep reinforcement learning architecture. Secondly, based on the multi-sensory fusion of Nokov-Lidar, an environmental state observation space under imperfect perception conditions is designed. Then, based on the grid world navigation map, a discretized action space is designed. At the same time, different state decision steps are set to improve the reward at each time step, so that the mobile robot will be penalized if it collides, but will receive a larger reward when reaching the target position during the navigation process. Finally, a premature collision prediction module is constructed based on a multi-layer perceptron to assist the multi-threaded asynchronous deep reinforcement learning architecture in learning a more robust collision avoidance strategy. This method can achieve sufficient mobile robot-environment interaction, maximize the exploration level of the robot action decision model in complex dynamic scenes, ensure that the mobile robot successfully avoids collisions while moving towards the target, and thus achieve better obstacle avoidance path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is the overall flow chart of the mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning proposed in the present invention;

[0066] Figure 2 Schematic diagram of local collision avoidance of a mobile robot based on global guidance path constraints in the method proposed in the present invention;

[0067] Figure 3 This is a structural diagram of the premature collision prediction module in the method proposed by the present invention;

[0068] Figure 4 This figure compares the obstacle avoidance trajectory results generated by different dynamic obstacle avoidance methods in detour scenarios using the method proposed in this invention and existing methods. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the following Figure 1-4 The present invention is further described in detail with specific implementation methods, so that the application can fully understand how to use technical means to solve technical problems and achieve technical effects and implement them accordingly.

[0070] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] like Figure 1 As shown, the present invention proposes a mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning, which specifically includes the following steps:

[0072] S1. Based on the PPO algorithm, a multi-task asynchronous parallel mechanism is added to build a multi-threaded asynchronous deep reinforcement learning architecture MAPPO, which separates different obstacle avoidance task scenarios and trains them simultaneously;

[0073] As a preferred embodiment, step S1 specifically includes:

[0074] S11. Use the Trust Region Policy Optimization algorithm TRPO to describe the strategy optimization problem of mobile robot path planning. The formula is expressed as:

[0075]

[0076] in, and π θ They are the old strategy before the strategy network is updated and the new strategy after the update; is the advantage function, calculated based on the generalized advantage estimation algorithm; X t 、a t Represent the state and action at time t respectively Express expectations, Represents the state-action trajectory distribution; In this embodiment, by adding constraints to limit the amount of change in the strategy, the state-action trajectory distribution can be approximately based on the old strategy, that is, setting

[0077] S12. Construct a loss function based on the PPO-Clip algorithm to simplify the optimization problem described by the TRPO algorithm; the constructed loss function The formula is expressed as:

[0078]

[0079] in, Represents the current policy π θ The action probability under the previous strategy The ratio of the action probabilities under t When (θ)∈[1-∈,1+∈], the function outputs Otherwise, the upper bound 1+∈ or the lower bound 1-∈ is output; in this embodiment, ∈=0.2 is taken, which is the most effective after experimental verification; in practical applications, in order to improve the sample utilization rate, the method of the present invention will be based on the old strategy The sampling trajectory updates the policy parameters multiple times, thereby alleviating the low sample efficiency problem of the same-policy deep reinforcement learning algorithm to a certain extent;

[0080] S13. Minimize the mean square error based on the gradient descent algorithm to update the value network of PPO-Clip; the formula is expressed as:

[0081]

[0082] Among them, ω is the value network update parameter; r t′ is the reward at time t′; V represents the reward that the robot will get starting from t′; ω (X t ) represents the state value; γ is a hyperparameter used to adjust the variance; in this embodiment, the value network is optimized based on the Adam optimizer, and the parameters θ and ω are respectively adjusted at the learning rate lr in each epoch. θ =3e-4 and lr ω =1e-3 update;

[0083] S14. Based on the PPO-clip strategy calculation, a multi-task asynchronous parallel mechanism is added to build the MAPPO architecture. In the MAPPO architecture, each Worker child node process randomly assigns a task scenario to the mobile robot for interaction and data collection. At the same time, the gradient value is calculated based on the interaction trajectory and sent to the Global master node.

[0084] Furthermore, by presetting obstacle avoidance scenarios of varying complexity, each process robot interacts in a different task environment, thereby increasing the task generalization capability of the overall algorithmic policy model and implicitly increasing the policy's exploration level. In the MAPPO architecture, data collection and gradient calculation are distributed across multiple workers. The Global node periodically receives the gradient values ​​submitted by each worker, updates the main network parameters based on the average gradient criterion, and distributes the latest parameters to other worker child nodes, ultimately implementing a multi-threaded asynchronous deep reinforcement learning mechanism.

[0085] S2. Based on the multi-sensor perception information of Nokov-Lidar (i.e., motion capture system-laser radar), a robot environment state observation space is constructed to provide environmental understanding and state estimation for the mobile robot, so as to accurately extract the motion and position characteristics of obstacles and the robot's self-state;

[0086] As a preferred embodiment, step S2 is specifically as follows:

[0087] Construct the robot environment state observation space, and the state observation sequence of each time step is recorded as:

[0088]

[0089] Among them, l t The 72 distance values ​​provided by the 360-degree lidar; v t A 2D vector representing the current rotational and translational velocity of the mobile robot; and It is a 2D navigation path constraint constructed based on the global guidance path (point-to-point distance between the starting position and the target position) and the robot's own state under Nokov global perception; t is the one-dimensional heading angle;

[0090] The three historical observation sequences are superimposed as the current input for each time step, namely:

[0091] X t =(x t-1 , x t-1 , x t );

[0092] Among them, X t Represents the state input at the current time t, x t-2 , x t-1 , x t They are the state observation sequences at time t-2, t-1, and t respectively; in this application, X t As the input to subsequent models and model loss functions, it not only considers the current state, but also takes into account the impact of historical states on the current state and causal relationships, so as to more effectively predict the movement of dynamic obstacles and accurately perceive collision threats.

[0093] S3. Based on Nokov’s global grid world navigation map, we design a discretized action space and set kinematic constraints for the mobile robot’s state update during the action decision process. Since each scene in the navigation dataset is discretized into a grid world navigation map, it is easy to obtain the global guidance path for the collision avoidance task under the premise of Nokov perception, such as Figure 2 As shown; therefore, this application also discretizes the action space, specifically:

[0094] The discretized motion of the designed robot is: in,

[0095]

[0096] v l and v w It is the reference value of translation speed and rotation speed set based on the environment;

[0097] Add the discretized action to the action space S, the formula is expressed as:

[0098]

[0099] The action space includes 28 actions including the stop action. In the action decision process, action constraints are set for the robot state update. That is, the formula for the robot state update within the time interval Δt is expressed as:

[0100]

[0101] in, and are the robot's position p at time t t The horizontal and vertical components of t is the one-dimensional heading angle; and are translational velocity and rotational velocity respectively; is the state vector at time t′ after the action is executed, Δt=t′-t.

[0102] S4. Design a navigation reward function to guide the mobile robot to make the best obstacle avoidance decisions in complex environments;

[0103] The goal of the autonomous obstacle avoidance model is to help the mobile robot respond quickly to environmental changes and ensure that it can robustly complete navigation tasks in complex dynamic scenarios, even if the robot safely navigates to the target location. Therefore, the reward design directly determines whether the robot can be effectively guided, and it plays a vital role in the robot's completion of the navigation task requirements. In order to enable the robot to continuously obtain rewards and avoid deviant behaviors such as greed and recklessness, the present invention sets different state penalties to improve the rewards at each time step, so that the mobile robot will be penalized if it collides, but will receive a larger reward when it reaches the target location during navigation; as a preferred embodiment, step S4 is specifically as follows:

[0104] Consider moving the robot at maximum speed And with The same direction, ideally traveled a desired distance The navigation reward function at each moment during this period is calculated as:

[0105]

[0106] in, is the time step penalty, which is used to encourage the robot to shorten the decision time; is the tangential motion reward function, which is used to constrain the robot to move along the direction of the global guidance path; is the normal motion reward function; Penalty for collision; To reach the reward;

[0107] The formula is:

[0108]

[0109] in, Indicates the movement distance of the robot in the tangent direction of the global guidance path.

[0110] The formula is:

[0111]

[0112] in, Represents the distance traveled by the robot in the normal direction of the global guidance vector; and They are and exist Normal vector The module of the projection;

[0113] The formula is expressed as:

[0114]

[0115] The formula is expressed as:

[0116]

[0117] Among them, B k (0≤k≤M) represents the position of the kth obstacle; p g is the target point position; r collision and r goal There are two sparse feedbacks used to adjust the robot's behavior to meet the physical hard constraints. In this embodiment, r collsion = -10 and r goal =25.

[0118] S5. Build a premature collision prediction module based on a multi-layer perceptron network to infer collision information from the perceptible environment, and assist the multi-threaded asynchronous deep reinforcement learning architecture in training a more reliable obstacle avoidance strategy.

[0119] In order to further improve the safety and robustness of autonomous obstacle avoidance of mobile robots in complex dynamic scenes, this application adds a premature collision prediction module to the navigation strategy, which is a multi-layer perceptron, such as Figure 3 As shown, it is supervised by a global guidance path to learn high-level feature representations to provide collision probabilities for all actions.

[0120] As a preferred embodiment, step S5 specifically includes:

[0121] S51, such as Figure 3 As shown, this application designs a premature collision module PCP based on a multi-layer perceptron, which converts the state X of each time step into t As the current input of PCP, calculate all possible actions ca at each time step t The collision probability PCP(ca t |X t ), each possible action is labeled as:

[0122] ca t ~PCP(ca t |X t );

[0123] S52. Use multi-label classification loss as the loss function for PCP module training The formula is as follows:

[0124]

[0125] in, is the distance from the robot's current position to its local track point g b The preferred action of ,the local track points are obtained by discretizing the global guidance path; yes The delta function at , which is supervised by the global guidance path;

[0126] S53. Introduce the premature collision prediction module into MAPPO to jointly learn the navigation strategy; that is, construct a joint learning loss function As the total loss function of MAPPO after the introduction of the PCP module, is a loss function constructed based on the PPO-Clip algorithm; the total loss function guides the model to learn a navigation strategy, resulting in an optimal obstacle avoidance strategy. In this embodiment, δ = 0.3 is set to control the strength of the premature collision prediction module's loss term. The premature collision prediction module can rapidly infer collision information from the perceived environment through high-level representation learning, which facilitates more robust obstacle avoidance strategy learning and ultimately achieves safer navigation.

[0127] Figure 4 The figure shows a comparison of the autonomous obstacle avoidance trajectory results of the proposed method and the classic APF method in a detour scenario. This comparison demonstrates that the proposed MAPPO method can output a reasonable strategy, enabling the mobile robot to successfully avoid moving obstacles and reach its target. Surprisingly, the APF method fails to achieve collision avoidance. This may be because the repulsive force of the obstacle is much less influential on the mobile robot than the attractive force of the target in this scenario, making it difficult for the APF algorithm to plan a detour path to avoid the obstacle.

[0128] In summary, the method of the present invention can achieve sufficient mobile robot-environment interaction, maximize the exploration degree of the robot action decision model in complex dynamic scenes, and ensure that the mobile robot successfully avoids collisions while moving towards the target.

[0129] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0130] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0131] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning, characterized by: The specific steps include: S1. Based on the PPO algorithm, a multi-task asynchronous parallel mechanism is added to build a multi-threaded asynchronous deep reinforcement learning architecture MAPPO, which separates different obstacle avoidance task scenarios and trains them simultaneously; S2. Based on the nokov-lidar multi-sensor perception information, a robot environment state observation space is constructed to provide environment understanding and state estimation for the mobile robot; S3. Based on Nokov's global grid world navigation map, design a discretized action space and set kinematic constraints for the mobile robot's state update during the action decision process; S4. Design a navigation reward function to guide the mobile robot to make the best obstacle avoidance decisions in complex environments; S5. Build a premature collision prediction module PCP based on a multi-layer perceptron to infer collision information from the perceptible environment, and combine it with the multi-threaded asynchronous deep reinforcement learning architecture MAPPO to learn and train the optimal obstacle avoidance strategy.

2. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 1 is characterized in that: Step S1 specifically includes: S11. Use the Trust Region Policy Optimization algorithm TRPO to describe the strategy optimization problem of mobile robot path planning. The formula is expressed as: in, and π θ They are the old strategy before the strategy network is updated and the new strategy after the update; is the advantage function, calculated based on the generalized advantage estimation algorithm; X t 、a t Represent the state and action at time t respectively Express expectations, Represents the distribution of state-action trajectories; S12. Construct a loss function based on the PPO-Clip algorithm to simplify the optimization problem described by the TRPO algorithm; the constructed loss function The formula is expressed as: in, Represents the current policy π θ The action probability under the previous strategy The ratio of the action probabilities under t When (θ)∈[1-∈,1+∈], the function outputs Otherwise, output the upper bound 1+∈ or the lower bound 1-∈; S13. Minimize the mean square error based on the gradient descent algorithm to update the value network of PPO-Clip; the formula is expressed as: Among them, ω is the value network update parameter; r t′ is the reward at time t′; V represents the reward that the robot will get starting from t′; ω (X t ) represents the state value; γ is a hyperparameter used to adjust the variance; S14. Based on the PPO-clip strategy calculation, a multi-task asynchronous parallel mechanism is added to build the MAPPO architecture. In the MAPPO architecture, each Worker child node process randomly assigns a task scenario to the mobile robot for interaction and data collection. At the same time, the gradient value is calculated based on the interaction trajectory and sent to the Global master node.

3. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 1 is characterized in that: Step S2 is specifically as follows: Construct the robot environment state observation space, and the state observation sequence of each time step is recorded as: Among them, l t It is the distance point cloud data provided by the laser radar scanning one week; t A 2D vector representing the rotational and translational velocities of the mobile robot at time t; and It is a 2D navigation path constraint constructed based on the global guidance path and the robot's own state under Nokov global perception; t is the one-dimensional heading angle; Superimpose three consecutive historical observation sequences as the current state input for each time step, that is: X t =(x t-2 ,x t-1 ,x t ); Among them, X t Represents the state input at the current time t, x t-2 , x t-1 , x t are the state observation sequences at time t-2, t-1, and t respectively.

4. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 1 is characterized in that: The action space designed in step S3 is specifically: The discretized motion of the designed robot is: in, v l and v w It is the reference value of translation speed and rotation speed set based on the environment; Add the discretized action to the action space S, the formula is expressed as: The action space contains 28 actions including stop actions.

5. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 4 is characterized in that: In step S3, during the action decision process, action constraints are set for the robot state update, that is, the formula for the robot state update within the time interval Δt is expressed as: in, and are the robot's position p at time t t The horizontal and vertical components of t is the one-dimensional heading angle; and are translational velocity and rotational velocity respectively; is the state vector at time t′ after the action is executed, Δt=t′-t.

6. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 1 is characterized in that: Step S4 is specifically as follows: Consider moving the robot at maximum speed And with The same direction, ideally traveled a desired distance The navigation reward function at each moment during this period is calculated as: in, is the time step penalty, which is used to encourage the robot to shorten the decision time; is the tangential motion reward function, which is used to constrain the robot to move along the direction of the global guidance path; is the normal motion reward function; Penalty for collision; To reach the reward; The formula is: in, Indicates the movement distance of the robot in the tangent direction of the global guidance path. The formula is: in, Represents the distance traveled by the robot in the normal direction of the global guidance vector; and They are and exist Normal vector The module of the projection; The formula is expressed as: The formula is expressed as: Among them, B k (0≤k≤M) represents the position of the kth obstacle; p g is the target point position; r collision and r goal There are two types of sparse feedback used to adjust the robot's behavior to meet physical hard constraints.

7. The mobile robot autonomous obstacle avoidance method based on multi-threaded asynchronous deep reinforcement learning according to claim 1 is characterized in that: Step S5 specifically includes: S51, based on the multi-layer perceptron design premature collision module PCP, the state of each time step X t As the current input of PCP, calculate all possible actions ca at each time step t The collision probability PCP(ca t |X t ), each possible action is labeled as: that t ~PCP(as t |X t ); S52. Use multi-label classification loss as the loss function for PCP module training The formula is as follows: in, is the distance from the robot's current position to its local track point g b The preferred action of ,the local track points are obtained by discretizing the global guidance path; yes The delta function at , which is supervised by the global guidance path; S53. Introduce the premature collision prediction module into MAPPO to jointly learn the navigation strategy; that is, construct a joint learning loss function As the total loss function of MAPPO after the introduction of the PCP module, is a loss function constructed based on the PPO-Clip algorithm; the total loss function guides the model to learn the navigation strategy and obtain the optimal obstacle avoidance strategy.

Images