Autonomous obstacle avoidance method for mobile robot

Abstract

The invention discloses an autonomous obstacle avoidance method for a mobile robot. The method comprises the steps that a simulation environment is built, and a state vector is obtained; constructing a deep reinforcement learning network; designing a total reward function; carrying out iterative training and optimization on the deep reinforcement learning network in static and dynamic scenes; and migrating the deep reinforcement learning network to a real mobile robot, and carrying out navigation test experiments in static and dynamic environments. According to the invention, by integrating the security value function and the security action projection module of the asymmetric obstacle Lyapunov function, a'decision-making layer-execution layer 'dual security guarantee mechanism is constructed, so that the output and execution of dangerous actions are completely eradicated from the source, and the security and reliability of the system are remarkably improved. Smooth and low-cost migration from simulation to real objects is achieved, a complete, efficient and practical autonomous obstacle avoidance solution of the mobile robot is formed, and the method has good engineering application value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent robot control and autonomous navigation, and particularly relates to a mobile robot autonomous obstacle avoidance method. BACKGROUND

[0002] Mobile robots are increasingly widely used in various fields, including industrial logistics, medical treatment, etc. An intelligent mobile robot obstacle avoidance system based on reinforcement learning has become a hot topic in this field. The reinforcement learning method guides the mobile robot to autonomously learn an efficient path planning strategy through a reward function. An active obstacle avoidance system is a core module of autonomous navigation of the mobile robot, and aims to enable the mobile robot to safely and efficiently avoid obstacles in a static or dynamic environment and complete target path planning.

[0003] In the conventional technology, a plurality of sensors are fused to obtain real-time self-position information of the intelligent mobile robot, and a working area, a starting point and an end point of the mobile robot are specified to plan an optimal control strategy. However, this centralized obstacle avoidance algorithm has the disadvantages of recognition limitation of a traditional fixed feature library, low recognition accuracy, etc., and thus is not suitable for the intelligent mobile robot to work in a complex dynamic environment. In comparison, the active obstacle avoidance strategy based on deep reinforcement learning relies on the ROS platform, and can better solve the obstacle avoidance problem of the mobile robot in an unknown and complex dynamic environment.

[0004] In addition, the conventional obstacle avoidance algorithms, such as A*, DWA and RRT, have the problems of inaccurate prediction of dynamic obstacles, and large path planning time error of the mobile robot in the obstacle avoidance process. A* depends on a known static map and cannot handle dynamic obstacles, and depends on the quality of a heuristic function. The DWA algorithm only considers the states in a local window and is prone to fall into a local optimal trap, and has limited prediction ability for obstacles in a complex environment. The RRT algorithm has large computational resource consumption and is difficult to solve the problem of narrow channel, and the path planning is non-optimal. Although deep reinforcement learning (such as the SAC algorithm) performs well in path planning, it lacks an explicit constraint mechanism for state variables in the learning process, which may cause the mobile robot to cross the safety boundary (such as too close to the obstacle and too fast, etc.) in the training or execution process, and has a collision risk. In summary, the existing methods mostly rely on a penalty term in the reward function to indirectly guide the intelligent agent, and lack theoretical safety guarantee. Therefore, there is an urgent need in the field for a mobile robot autonomous obstacle avoidance method capable of strictly guaranteeing full-state safety constraints in the reinforcement learning process, to solve the problems of insufficient safety and poor reliability of the existing technology in a dynamic and complex environment. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a mobile robot autonomous obstacle avoidance method.

[0006] In order to achieve the above object, the autonomous obstacle avoidance method of the mobile robot provided by the application comprises the following steps in sequence:

[0007] 1) A ROS / Gazebo simulation environment is built in Linux, and state information of the mobile robot and state information of the obstacles are obtained through the simulation environment, and then an integrated state vector is obtained based on the above two state information;

[0008] 2) A deep reinforcement learning network is constructed;

[0009] 3) A total reward function is designed according to the obstacle avoidance task of the mobile robot;

[0010] 4) Based on the integrated state vector obtained in step 1) and the total reward function obtained in step 3), the deep reinforcement learning network obtained in step 2) is iteratively trained and optimized in the ROS / Gazebo simulation environment under static and dynamic scenarios, and a trained deep reinforcement learning network is obtained;

[0011] 5) The trained deep reinforcement learning network obtained in step 4) is migrated to a real mobile robot, and navigation test experiments are carried out in static and dynamic environments respectively, so that a complete closed loop from simulation to real object is realized.

[0012] In step 1), the simulation environment comprises randomly distributed static obstacles, a plurality of dynamic obstacles moving along a preset path and a mobile robot carrying a simulated 2D laser radar; the static obstacles are used to simulate walls, boxes and other objects, the dynamic obstacles are used to simulate pedestrians or moving objects, and the model of the mobile robot is TurtleBot3; the size of the simulation environment is 10m*10m;

[0013] The state information of the mobile robot comprises its own radius r, its own position coordinates m[m x , m y ], motion speed v[v x , v y ], position coordinates n[n x , n y ] when reaching the target point and direction θ;

[0014] The mobile robot collects the point cloud of the surrounding environment through the laser radar; the relative position and posture of the laser radar are calibrated, and the coordinate system in which the laser radar is located is defined as the laser coordinate system; the projection direction of the laser radar is taken as the z-axis of the laser coordinate system, and the data timestamp is aligned; the rotation matrix and translation vector of the laser radar are set; a chessboard with known size is used, the position of the corner points in the point cloud is detected, and the laser radar is dynamically calibrated based on the edges and corner points, so that the direct alignment and joint processing of the point cloud are completed;

[0015] The obstacle's status information includes the obstacle's position coordinates [s] i x s i y Speed ​​[v] i x v i y ] and radius r i ;

[0016] Then, at time t, the state vector of the mobile robot is:

[0017] M t =[m x m y v x v y ,r,n x n y ,θ]

[0018] The state vector of the i-th obstacle under the observation state is:

[0019] N t =[s i x s i y v i x v i y r i ]

[0020] Finally, based on the two state vectors mentioned above, the integrated state vector St = [Mt, Nt] is obtained and output as the input to the subsequent deep reinforcement learning network.

[0021] In step 2), the deep reinforcement learning network is based on the improved SAC framework, including a policy network, a value network, and a safe action projection module. The policy network employs a stochastic policy and includes two sub-networks: mean and variance. Its input is the integrated state vector St output in step 1). The mean and variance sub-networks process the state vector St and output the mean and standard deviation of the action policy, respectively. Then, the original action a of the mobile robot is obtained through reparameterization trick sampling. raw =(v raw ω raw ), where v raw ω represents linear velocity. raw Representing angular velocity, finally the original motion a raw =(v raw ,ω raw The signals are transmitted to the safety action projection module and the value network, respectively.

[0022] The value network integrates the asymmetric barrier Lyapunov function into the definition of the value function, and constructs a safe state-action value function Q safe :

[0023] Q safe (s, a) = Q(s, a) - λ × V(z)

[0024] where V(z) is an asymmetric barrier Lyapunov function, and the parameters are set as p = 1, k ai = k bi = 0.5, and the safety weight coefficient λ is initialized as 0.8; z represents the distance offset between the current system state and the safety boundary of the barrier, z = d min - d safe , where d min is the minimum distance between the current mobile robot and the nearest barrier, and d safe is the set safety distance threshold; q is a weighted parameter of state safety risk, used to describe the safety risk sensitivity at different proximity levels.

[0025] The input of the value network is the integrated state vector St and the original action a raw = (v raw , ω raw ), and the output is the estimated value Q(St, a) of the state-action value, used to evaluate the safety of the original action a raw = (v raw , ω raw ) and guide the update of the policy network.

[0026] The safety action projection module constrains the original action a raw = (v raw , ω raw ) output by the policy network and projects it into the absolute allowed safety action space of the mobile robot physical platform, to obtain the projected safety action a safe of the mobile robot, as follows:

[0027] a safe = [clip(v raw , v min + δ, v max - δ), clip(w raw , w min + δ, w max - δ)]

[0028] where clip is a clipping function, v min , v max and w minw max is the speed and angular velocity boundary allowed by the hardware platform, and δ is the safety margin.

[0029] In step 3), the total reward function R total is composed of multiple sub-reward items, including the arrival reward, the progress reward, the speed reward, the direction penalty, the safety distance and the safety penalty.

[0030] The total reward function R total is expressed as:

[0031] R total = w1R q + w2R c + w3R v + w4R d + w5R s + w6R b

[0032] Wherein:

[0033] R q is the arrival reward of the mobile robot, which is set as the following function:

[0034]

[0035] Wherein, d p is the distance between the mobile robot and the target point, and c p is a custom distance.

[0036] R c is the progress reward, which is defined as the following function:

[0037] R c = (d previous - d current ) × k

[0038] Wherein, d current is the Euclidean distance between the mobile robot and the target point at the current time step; d previous is the Euclidean distance between the mobile robot and the target point saved at the last time step.

[0039] An adaptive progress reward coefficient is introduced:

[0040]

[0041] Wherein, N fail is the number of recent obstacle avoidance failures; N total is the total number of training rounds; and β is the exploration weight.

[0042] R v is the speed reward, which is set as the following function:

[0043]

[0044] where d safe is the set safety distance threshold; d min is the minimum distance from the mobile robot to the nearest obstacle; λ is a self-defined coefficient, ranging from 0 to 0.8; the safety distance threshold d safe satisfies the following function:

[0045]

[0046] where d base is the basic safety distance, set to within 0.6 m; v is the current linear speed of the mobile robot, v max is the maximum allowed linear speed; μ is a sensitivity coefficient, set to 0-0.5;

[0047] R d is the direction penalty, set as the following function:

[0048] R d = -α × |ω|

[0049] where α is a self-defined penalty coefficient, determined by experiment adjustment; ω is the angular speed of the mobile robot; the penalty coefficient is designed as:

[0050]

[0051] g i is the penalty parameter;

[0052] R s is the safety distance penalty, set as the following function:

[0053]

[0054] where ∈ is a smoothing constant; the adaptive smoothing constant is introduced as follows:

[0055] ∈ = 0.1 + γ × σ

[0056] where σ is the standard deviation of the laser radar distance measurement; γ is a noise sensitivity coefficient;

[0057] R b is the safety penalty, which is designed based on the asymmetric obstacle Lyapunov function V(z) and set as the following function:

[0058]

[0059] In the total reward function R total = w1R q + w2R c + w3Rv +w4R d +w5R5+w6R b In the above, the weights of each sub-reward item are set as [w1, w2, w3, w4, w5, w6] = [1.0, 2.0, 0.5, 0.1, 0.8, 0.5].

[0060] In step 4), based on the integrated state vector St obtained in step 1) and the total reward function R total , the deep reinforcement learning network obtained in step 2) is iteratively trained and optimized in a static and dynamic scenario in a ROS / Gazebo simulation environment. The method for obtaining the trained deep reinforcement learning network is:

[0061] In the dynamic scenario, the simulation environment includes randomly distributed static obstacles and multiple dynamic obstacles moving along a preset path; the initial pose of the mobile robot, the target point, and the motion trajectory of the dynamic obstacles are randomly generated; the training process is as follows: the mobile robot perceives the environment through the laser radar to obtain the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(v raw ,ω raw ); the original action a raw =(v raw ,ω raw ) is input into the value network and the safety action projection module at the same time; the value network uses the safety state-action value function Q safe to evaluate the safety of the action and guide the policy network to update; the safety action projection module projects the original action a raw =(v raw ,ω raw ) into the safety action space to obtain the safety action a safe ; the safety action a safe is used for environment interaction, and the reward is calculated based on the total reward function R total ; the mobile robot adjusts the strategy according to the reward value until it successfully avoids obstacles or reaches the target point; each round of training lasts at most 500 steps, and if collision occurs or the target point is reached, the training is terminated in advance; the mobile robot is considered to have successfully simulated once it moves from the starting point without collision to the target point, and then the starting point and the target point are reset for a new round of simulation. The training is repeated until the policy network converges.

[0062] In the static scenario, the simulation environment includes a fixed map, fixed obstacles, a fixed starting point, and a fixed target point; the training process is as follows: the mobile robot perceives the environment through the laser radar to obtain the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(vraw , ω raw ); original action a raw =(v raw , ω raw ) is processed by a safe action projection module to obtain a safe action a safe ; the safe action a safe is used for environmental interaction, and a reward is calculated based on a total reward function R total ; if a collision occurs or a target point is reached, the starting point of the mobile robot is reset, but the current policy parameters are retained for continued training; through fixed scene training, the deep reinforcement learning network is strengthened to learn a basic obstacle avoidance strategy;

[0063] Finally, a deeply trained and parameter-optimized deep reinforcement learning network is obtained.

[0064] In step 5), the trained deep reinforcement learning network obtained in step 4) is migrated to a real mobile robot, and navigation test experiments are performed in static and dynamic environments respectively, so that a complete closed loop from simulation to real object is realized.

[0065] In a static environment, an experimental scene is constructed, and obstacles are laid out; a piece of open land is selected, and randomly distributed cylinders, cubes and rhombic columns are set as static obstacles on the open land, and then a benchmark path planning is performed on the mobile robot, and a target point is set; the parameters of the deep reinforcement learning network are adjusted, and the weights of each sub-reward item in the total reward function are adjusted to verify the control response at different speeds; the parameters include the weights of the policy network, the entropy temperature coefficient, the weights of the value network, the safety weight coefficient λ, the weighted parameter q of the state safety risk in the asymmetric obstacle Lyapunov function, and the weights of each sub-reward item in the total reward function; the basic obstacle avoidance capability is verified in the controlled static environment, dynamic elements are gradually introduced, an experimental scene is constructed, dynamic obstacles are set, and a random number of human-shaped and mobile robots are randomly moved with a speed of 0-4m / s; the mobile robot obtains point cloud data through a laser radar, predicts the dynamic obstacles, and generates a local map; the policy network and the value network are used for hierarchical control debugging; when the mobile robot collides with the dynamic obstacle, a test is ended, and human-machine interaction scene training data is counted; the indicators of the mobile robot are evaluated, the accuracy of trajectory prediction and the handling of sudden obstacles in dynamic response are evaluated, and the system performance is comprehensively evaluated.

[0066] The mobile robot autonomous obstacle avoidance method provided by the application has the following beneficial effects:

[0067] 1. By integrating the safety value function of the asymmetric obstacle Lyapunov function and the safety action projection module, a "decision layer-execution layer" double safety guarantee mechanism is constructed, which fundamentally eliminates the output and execution of dangerous actions and significantly improves the safety and reliability of the system.

[0068] 2. The reward function containing multiple adaptive parameters is designed, so that the system can dynamically adjust and optimize the goal according to the environmental complexity and historical experience, effectively avoiding local optimization, not only solving the obstacle avoidance strategy of mobile robots in static environment, but also greatly improving the success rate of autonomous obstacle avoidance of mobile robots in complex dynamic environment.

[0069] 3. Smooth and low-cost migration from simulation to physical object is realized, forming a complete, efficient and practical autonomous obstacle avoidance solution for mobile robots, which has good engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The autonomous obstacle avoidance method for mobile robots provided by the present application is provided.

[0071] Figure 2 The iteration process flowchart of "test-evaluate-parameter fine-tuning" in the present application.

[0072] Figure 3 The deep reinforcement learning network structure schematic diagram provided by the present application. DETAILED DESCRIPTION

[0073] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0074] As shown in the drawings, Figure 1 The autonomous obstacle avoidance method for mobile robots provided by the present application includes the following steps in sequence:

[0075] 1) A ROS / Gazebo simulation environment is built in Linux, and the state information of the mobile robot and the state information of the obstacles are obtained through the simulation environment, and then the integrated state vector is obtained based on the above two state information;

[0076] The simulation environment includes randomly distributed static obstacles, a plurality of dynamic obstacles moving along a predetermined path, and a mobile robot or a moving object carrying a simulated 2D laser radar; the static obstacles are used to simulate walls, boxes and other objects, the dynamic obstacles are used to simulate pedestrians, and the model of the mobile robot is TurtleBot3; the size of the simulation environment is 10m*10m;

[0077] The state information of the mobile robot includes its own radius r, its own position coordinates m[m x ,m y], the moving speed v [v x ,v y ], the position coordinates n [n x ,n y ] and the direction θ when reaching the target point;

[0078] The mobile robot collects point clouds of the surrounding environment through the laser radar; the relative position and posture of the laser radar are calibrated, and the coordinate system in which the laser radar is located is defined as the laser coordinate system; the projection direction of the laser radar is taken as the z-axis of the laser coordinate system, and the data timestamp is aligned; the rotation matrix and translation vector of the laser radar are set; a chessboard with a known size is used to detect the position of the corner points in the point cloud, and the laser radar is dynamically calibrated based on the edges and corner points, so that the direct alignment and joint processing of the point cloud are completed;

[0079] The state information of the obstacle includes the position coordinates [s i x ,s i y ], the speed [v i x ,v i y ] and the radius r i ;

[0080] Then, the state vector of the mobile robot at time t is:

[0081] M t =[m x ,m y ,v x ,v y ,r,n x ,n y ,θ]

[0082] The state vector of the i-th obstacle in the observation state is:

[0083] N t =[s i x ,s i y ,v i x ,v i y ,r i ]

[0084] Finally, the integrated state vector St=[M t ,N t ] is obtained based on the above two state vectors and is output as the input of the subsequent deep reinforcement learning network.

[0085] 2) Construct as Figure 3The deep reinforcement learning network shown;

[0086] The deep reinforcement learning network is based on an improved SAC (Soft Actor-Critic) framework, including a policy network (Actor), a value network (Critic) and a safe action projection module; wherein the policy network adopts a stochastic policy, and contains two sub-networks of mean and variance; the input is the integrated state vector St output in step 1), the state vector St is processed by using the two sub-networks of mean and variance, and the mean and standard deviation of the action policy are output respectively, and then the original action a of the mobile robot is obtained by sampling through the reparameterization technique raw =(v raw ,ω raw ), wherein v raw represents linear velocity, and ω raw represents angular velocity, and finally the original action a raw =(v raw ,ω raw ) is transmitted to the safe action projection module and the value network respectively;

[0087] In addition, in order to dynamically balance exploration and utilization, the entropy temperature coefficient α of the SAC algorithm in the policy network is set as a trainable adaptive parameter. By minimizing the loss function J(α)=α(logπ(a|s)+H_target), wherein H_target is the target entropy, representing the expected policy entropy level, and logπ(a|s) is the logarithmic probability of the policy π taking action a in state s, the SAC algorithm can automatically increase the entropy temperature coefficient α in the early stage of training to encourage exploration, and reduce the entropy temperature coefficient α after the policy matures to encourage utilization, thereby significantly improving the learning efficiency and final performance in a complex environment.

[0088] The value network integrates an asymmetric barrier Lyapunov function (ABLF) into the definition of the value function, and constructs a safe state-action value function Q safe :

[0089] Q safe (s,a)=Q(s,a)-λ×V(z)

[0090] V(z) is an asymmetric barrier Lyapunov function, wherein the parameters are set as p=1, k ai =k bi =0.5, and the safety weight coefficient λ is initialized as 0.8; z represents the distance offset of the current system state and the safety boundary of the obstacle, z=d min -d safe , wherein d mind is the minimum distance between the current mobile robot and the nearest obstacle safe is a set safety distance threshold. When z > 0, it means the mobile robot is in the safety region; when z ≤ 0, it means the mobile robot has entered or crossed the safety boundary, and there is a risk of collision; q is a weighted parameter of state safety risk, which is used to describe the sensitivity of safety risk at different proximity, and can be adjusted adaptively according to the complexity of the environment, for example, increasing the weighted parameter q of state safety risk in a dynamic obstacle dense environment to improve the sensitivity of risk perception; reducing the weighted parameter q of state safety risk in an open environment to avoid excessive conservatism.

[0091] The safety state-action value function Q safe is based on the traditional value function Q(s, a), and an explicit penalty term λ · V(z) related to the state safety boundary is subtracted, so that the value network not only considers the long-term benefits when evaluating the original action, but also has a forward-looking safety risk awareness. By minimizing the Bellman error based on the safety state-action value function Q safe , the value network learns to preferentially deny high-risk actions, thereby guiding the policy network from the root of value to "not want" to output dangerous actions that exceed the boundary, so as to avoid the policy network learning dangerous strategies that may lead to state exceeding the boundary (such as being too close to the obstacle); during the training process, the value network uses the safety state-action value function Q safe to calculate the target value and update the network parameters; the policy network optimizes the policy according to the safety state-action value function Q safe to maximize the expected value.

[0092] The input of the value network is the integrated state vector St and the original action a raw = (v raw , ω raw ), and the output is the estimated value Q(St, a) of the state-action value, which is used to evaluate the safety of the original action a raw = (v raw , ω raw ) and guide the policy network to update;

[0093] The safety action projection module serves as a hard constraint layer to solve the dangerous actions that the policy network may output or unreasonable actions that the policy network outputs in unknown states. It constrains the original action a raw = (v raw , ω raw ) output by the policy network and projects it into the absolute allowed safety action space of the mobile robot physical platform to obtain the projected safety action a safe of the mobile robot, and the formula is as follows:

[0094] asafe = [clip(v raw , v min + δ, v max - δ), clip(w raw , w min + δ, w max - δ)]

[0095] where clip is a clipping function, v min , v max and w min , w max are the speed, angular speed bounds allowed by the hardware platform, and δ is the safety margin. The projected safe action a safe will be the input of the reward function calculation in step 3) and used for environment interaction in step 4). According to the physical limits of the TurtleBot3 model, in this invention, the linear speed bounds are set as [v min , v max ]= [-0.2, 0.8] m / s, the angular speed bounds are set as [ω min , ω max ]= [-2.0, 2.0] rad / s, and the safety margin δ is set as 0.05.

[0096] This mechanism constitutes the ultimate guarantee at the hardware execution level, ensuring that the system “cannot” perform out-of-bound actions, even if the policy network outputs dangerous instructions.

[0097] 3) Design a total reward function according to the obstacle avoidance task of the mobile robot;

[0098] The invention designs a total reward function R total to guide the mobile robot to learn the obstacle avoidance strategy in reinforcement learning training, which is composed of multiple sub-reward items, including the arrival target reward, progress reward, speed reward, direction penalty, safety distance, and safety penalty;

[0099] The expression of the total reward function R total is as follows:

[0100] R total = w1R q + w2R c + w3R v + w4R d + w5R s + w6R b

[0101] wherein:

[0102] R q is the arrival target reward of the mobile robot, which is set as the following function:

[0103]

[0104] where d p is the distance between the mobile robot and the target point, c p is a custom distance. When the mobile robot successfully reaches the target point, a base reward value of +300 is given immediately. If the mobile robot collides with an obstacle, a base penalty value of -50 is given to force the avoidance of dangerous behavior through negative incentives. The base reward value (+300) and the base penalty value (-50) of the target reaching reward have the highest priority and directly determine the round termination condition. Through a large numerical difference, the state of the successful and failed obstacle avoidance of the mobile robot is clearly distinguished.

[0105] R c is the progress reward, defined as the following function:

[0106] R c = (d previous - d current ) x k

[0107] where d current is the Euclidean distance between the mobile robot and the target point at the current time step; d previous is the Euclidean distance between the mobile robot and the target point saved at the last time step. When it is a positive difference, it indicates that the mobile robot is approaching the target point and the distance is shortened, and the mobile robot is given a positive reward; when it is a negative difference, it indicates that the mobile robot is moving away from the target point and the distance is increasing, and the mobile robot is given a penalty. When the difference is 0, it indicates that the mobile robot does not move or moves along a direction perpendicular to the target point, and at this time there is no reward or penalty. The fixed progress reward coefficient may lead to local optimization, such as circling a fixed path, so an adaptive progress reward coefficient is introduced:

[0108]

[0109] where N fail is the number of recent obstacle avoidance failures; N total is the total number of training rounds; and β is the exploration weight. When the recent failure rate increases, the progress adaptive reward coefficient is increased to encourage a more aggressive forward strategy to break through the local optimum; when the failure rate decreases, the default value is restored to avoid excessive risk-taking.

[0110] R v is the speed reward, set as the following function:

[0111]

[0112] where d safe is a set safety distance threshold; d minis the minimum distance from the mobile robot to the nearest obstacle; λ is a self-defined coefficient, ranging from 0 to 0.8. When d min > d safe , the mobile robot is in a safe area, and the speed reward takes effect. The mobile robot actively slows down when approaching the obstacle to avoid collision risks caused by high-speed movement. In the safe area, the reward is positively correlated with the speed, encouraging the mobile robot to quickly approach the target point; in the dangerous area, the speed reward is cancelled to prevent the mobile robot from ignoring obstacle avoidance for the sake of speed reward. At the same time, negative speed is allowed, and if the mobile robot produces meaningless backward behavior during training, it will obtain a negative reward. To avoid the fixed safety distance threshold being too conservative in narrow passages and not sensitive enough in open areas, an adaptive coefficient design is introduced to set the safety distance threshold d safe satisfies the following function:

[0113]

[0114] where d base is the basic safety distance, set to within 0.6 m; v is the current linear speed of the mobile robot, v max is the maximum allowed linear speed; μ is the sensitivity coefficient, set to 0-0.5. When the mobile robot moves at high speed, the safety distance threshold d safe is automatically increased to reserve a longer braking distance.

[0115] R d is the direction penalty, which is a penalty mechanism to avoid unnecessary turning or excessive turning, set as the following function:

[0116] R d = -α × |ω|

[0117] where α is a self-defined penalty coefficient, determined through experiments, used to adjust the influence of turning behavior on the total reward function; ω is the angular speed of the mobile robot. The direction penalty R d is only related to the turning amplitude and is independent of the direction. When the mobile robot turns at a higher speed, the penalty is more severe, thereby encouraging the mobile robot to reduce unnecessary turning and maintain smooth movement. To prevent the fixed penalty coefficient from suppressing necessary emergency turning, an adaptive coefficient design is added, and the penalty coefficient is designed as:

[0118]

[0119] When the minimum distance d min from the mobile robot to the nearest obstacle is ≤ 0.3 m, the penalty coefficient is increased to g1, set to 0.7-1, to force a large turning to quickly move away from the obstacle; when 0.3 m < d min<0.7, keep the default penalty coefficient g2 as 0.4~0.7, balance steering and progress; in the safety region, i.e., the minimum distance d of the mobile robot to the nearest obstacle min ≥0.7m, reduce the penalty coefficient g3 as 0~0.4, allow flexible path optimization.

[0120] R s is the safety distance penalty, set as the following function:

[0121]

[0122] where ∈ is a smoothing constant to avoid zero denominator and adjust the steepness of the penalty curve; set d min + ∈ to ensure that even if the minimum distance d min of the mobile robot to the nearest obstacle approaches 0, the value of the safety distance penalty will not be infinite, to ensure that the safety distance penalty range is controllable. The safety distance penalty formula quantifies the proximity of the mobile robot to the obstacle as a penalty value through an inverse proportional function, with the core goal of optimizing the path while ensuring safety. By high penalty, the mobile robot is forced to actively slow down or adjust the path when approaching the obstacle. Considering that a fixed smoothing constant may be unstable in a noisy environment, an adaptive smoothing constant is introduced as follows:

[0123] ∈ = 0.1 + γ × σ

[0124] where σ is the standard deviation of the laser radar distance measurement; γ is the noise sensitivity coefficient. In a high-noise environment, increase the smoothing constant ∈ to reduce the sensitivity of the penalty to noise. In a low-noise environment, reduce the smoothing constant ∈ to enhance the near-distance obstacle avoidance response.

[0125] R b is the safety penalty, which is designed based on the asymmetric obstacle Lyapunov function V(z) to ensure that the mobile robot avoids dangerous states during training, set as the following function:

[0126]

[0127] In the total reward function R total = w1R q + w2R c + w3R v + w4R d + w5R s + w6R bIn the above, the weights of each sub-reward item are set as [w1, w2, w3, w4, w5, w6] = [1.0, 2.0, 0.5, 0.1, 0.8, 0.5], wherein the reaching target reward function is the core priority function, and the speed reward, safety distance penalty and direction penalty all adopt the progressive guidance mechanism.

[0128] 4) Based on the integrated state vector obtained in step 1) and the total reward function obtained in step 3), the deep reinforcement learning network obtained in step 2) is iteratively trained and optimized in a static and dynamic scenario in a ROS / Gazebo simulation environment, and a trained deep reinforcement learning network is obtained;

[0129] In the dynamic scenario, the simulation environment includes randomly distributed static obstacles and multiple dynamic obstacles moving along a preset path; the initial pose of the mobile robot, the target point and the motion trajectory of the dynamic obstacles are randomly generated; the training process is as follows: the mobile robot perceives the environment through the laser radar to obtain the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(v raw , ω raw ); the original action a raw =(v raw , ω raw ) is input into the value network and the safety action projection module at the same time; the value network uses the safety state-action value function Q safe to evaluate the safety of the action and guide the policy network to update; the safety action projection module projects the original action a raw =(v raw , ω raw ) into the safety action space to obtain the safety action a safe ; the safety action a safe is used for environment interaction, and the reward is calculated based on the total reward function R totoal ; the mobile robot adjusts the strategy according to the reward value until it successfully avoids obstacles or reaches the target point; each round of training lasts at most 500 steps, and if collision occurs or the target point is reached, the training is terminated in advance; the mobile robot is considered to have successfully simulated once it moves from the starting point to the target point without collision, and then the starting point and the target point are reset for a new round of simulation, and the training is repeated until the policy network converges;

[0130] In the static scenario, the simulation environment includes a fixed map, fixed obstacles (such as walls), a fixed starting point and a target point; the training process is as follows: the mobile robot perceives the environment through the laser radar to obtain the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(v raw , ω raw);original action a raw =(v raw , ω raw ) is processed by a safe action projection module to obtain a safe action a safe ; the safe action a safe is used for environmental interaction, and a reward is calculated based on a total reward function R totoal ; if a collision occurs or a target point is reached, the starting point of the mobile robot is reset, but the current policy parameters are retained for continued training; through fixed scene training, the deep reinforcement learning network is strengthened to learn a basic obstacle avoidance strategy;

[0131] Finally, a deep reinforcement learning network that is fully trained and has optimized parameters is obtained.

[0132] In the training process of the present application, an Adam optimizer is used, and the learning rate of the policy network and the value network is set to 3e-4. The experience replay pool capacity is 100000. Each round of training lasts a maximum of 500 steps, and the robot is terminated in advance if a collision occurs or the target point is reached. After about 500,000 steps of training, the policy network converges successfully. In 100 random tests, the obstacle avoidance success rate reached more than 90%, and no safety distance violation occurred, i.e., the distance to the obstacle was less than 0.3 meters.

[0133] 5) The trained deep reinforcement learning network obtained in step 4) is migrated to a real mobile robot, and navigation test experiments are conducted in static and dynamic environments, respectively, so that a complete closed loop from simulation to reality is achieved, and the deep reinforcement learning network can not only be applied in a simulation environment, but also be applicable in an actually built mobile robot.

[0134] In a static environment, the experimental scene is constructed, and the obstacles are laid out; a piece of open land is selected, and random distribution of cylinders, cubes and rhombic columns are set as static obstacles on the open land, and then the benchmark path planning is performed for the mobile robot, and the target point is set; the parameters of the deep reinforcement learning network are adjusted, and the weights of each sub-reward item in the total reward function are adjusted to verify the control response at different speeds; the parameters include the weights of the policy network, the entropy temperature coefficient, the weights of the value network, the safety weight coefficient λ, the weighted parameter q of the state safety risk in the asymmetric obstacle Lyapunov function, and the weights of each sub-reward item in the total reward function; the basic obstacle avoidance capability is verified in the controlled static environment, and dynamic elements are gradually introduced to construct the experimental scene, set dynamic obstacles, and use an indefinite number of human-shaped and mobile robots to move randomly, and the speed is set to 0-4m / s; the mobile robot obtains the point cloud data through the laser radar, predicts the dynamic obstacles, and generates a local map; the hierarchical control debugging is performed through the policy network and the value network; when the mobile robot collides with the dynamic obstacle, a test is ended, and the human-machine interaction scene training data is counted; the indicators of the mobile robot are evaluated, the accuracy of trajectory prediction and the handling of sudden obstacles in dynamic response are evaluated, and the system performance is comprehensively evaluated. As shown in Figure 2 , through the iterative process of "test-evaluation-parameter fine-tuning", the performance of the deep reinforcement learning network is continuously optimized, and finally the mobile robot can also stably and safely complete the autonomous obstacle avoidance task in a real complex scene, which marks the realization of a complete closed loop from simulation to real object.

[0135] In step 5), the real mobile robot hardware system builds a complete physical platform corresponding to the algorithm framework with the main control unit as the core. The platform is composed of three parts of environment perception, decision control and action execution, realizing seamless connection from simulation to real object.

[0136] The environment perception device serves as the information input source of the system, providing real-time state vectors for the deep reinforcement learning network. Among them, the laser radar and the IMU / odometer work together: the laser radar scans the surrounding environment to generate point cloud data to detect the distance and contour of the obstacle; the IMU / odometer provides the acceleration, angular velocity and pose information of the robot itself, which together constitute the environment and self-state perception required by the algorithm.

[0137] The main control unit is the core of the whole system for calculation and decision, which is responsible for the deployment and operation of the trained deep reinforcement learning network. In this unit, the policy network and the value network collaborate to infer based on the perception information. The value network evaluates the safety of actions by fusing ABLF (Asymmetric Barrier Lyapunov Function), and the policy network outputs the original action command accordingly, which is processed by the safe action projection module to ensure that the command is always within the safety boundary of the physical platform. This unit realizes the closed-loop calculation of perception, safe decision and action projection.

[0138] The execution system is the final execution mechanism of the control command. The motor driver receives the linear and angular velocity commands from the main control unit that have been projected safely, and drives the chassis motor to move, thereby completing the navigation and obstacle avoidance actions of the mobile robot. Its physical performance parameters are strictly consistent with the constraint boundary of the safe action projection module in the simulation environment, which constitutes the final safety barrier of the hardware ontology.

[0139] In addition, the power module provides stable energy for all modules to ensure the continuous operation of the system. The communication interface is used to realize data exchange with the host computer, supporting online debugging of algorithms and monitoring of system status.

[0140] Through the close coupling of the above hardware modules, the deep reinforcement learning network in this method can be stably and safely run on a real mobile robot, forming a complete autonomous obstacle avoidance closed loop of "environment perception → safe decision → reliable execution".

Claims

1. A mobile robot autonomous obstacle avoidance method, characterized in that: The autonomous obstacle avoidance method of the mobile robot comprises the following steps performed in sequence: 1) A ROS / Gazebo simulation environment is built in Linux, and state information of the mobile robot and state information of the obstacles are obtained through the simulation environment, and then an integrated state vector is obtained based on the above two state information; 2) a deep reinforcement learning network is constructed; 3) a total reward function is designed according to the obstacle avoidance task of the mobile robot; 4) based on the integrated state vector obtained in step 1) and the total reward function obtained in step 3), the deep reinforcement learning network obtained in step 2) is iteratively trained and optimized in the ROS / Gazebo simulation environment under static and dynamic scenarios to obtain a trained deep reinforcement learning network; 5) the trained deep reinforcement learning network obtained in step 4) is migrated to a real mobile robot, and navigation test experiments are performed in static and dynamic environments respectively, so that a complete closed loop from simulation to real object is realized.

2. The method of claim 1, wherein: In step 1), the simulation environment includes randomly distributed static obstacles, a plurality of dynamic obstacles moving along a predetermined path, and a mobile robot carrying a simulated 2D laser radar; the static obstacles are used to simulate walls, boxes and other objects, the dynamic obstacles are used to simulate pedestrians or moving objects, and the model of the mobile robot is TurtleBot3; the size of the simulation environment is 10m*10m; The state information of the mobile robot includes its own radius r, its own position coordinates m[m x , m y ], movement speed v[v x , v y ], position coordinates n[n x , n y ] when reaching the target point, and direction θ; The mobile robot collects the point cloud of the surrounding environment through the laser radar; the relative position and attitude of the laser radar are calibrated, and the coordinate system in which the laser radar is located is defined as the laser coordinate system; the projection direction of the laser radar is taken as the z-axis of the laser coordinate system, and the data timestamp is aligned; the rotation matrix and translation vector of the laser radar are set; a chessboard with a known size is used, the position of the corner points in the point cloud is detected, and the laser radar is dynamically calibrated based on the edges and corner points, thereby completing the direct alignment and joint processing of the point cloud; The state information of the obstacle includes position coordinates [s i x ,s i y ] of the obstacle, speed [v i x ,v i y ] and radius r i ; Then, the state vector of the mobile robot at time t is: M t = [m x , m y , v x , v y , r, n x , n y , θ] The state vector of the i-th obstacle in the observation state is: N t = [s i x , s i y , v i x , v i y , r i ] Finally, the integrated state vector St = [M t , N t ] is obtained based on the above two state vectors and output as the input of the subsequent deep reinforcement learning network.

3. The method of claim 1, wherein: In step 2), the deep reinforcement learning network is based on an improved SAC framework, including a policy network, a value network and a safety action projection module; wherein the policy network adopts a stochastic policy, including two sub-networks of mean and variance; the input is the integrated state vector St output in step 1), the state vector St is processed by the two sub-networks of mean and variance to output the mean and standard deviation of the action policy respectively, and then the original action a of the mobile robot is obtained by sampling through the reparameterization technique raw = (v raw , ω raw ), wherein v raw represents linear velocity, ω raw represents angular velocity, and finally the original action a raw = (v raw , ω raw ) is transmitted to the safety action projection module and the value network respectively; The value network incorporates an asymmetric barrier Lyapunov function into the definition of the value function, constructing a safe state-action value function Q safe : Q safe (s, a) = Q(s, a) - λ x V(z) wherein V(z) is an asymmetric barrier Lyapunov function, wherein the parameters are set as p = 1, k ai = k bi = 0.5, and the safety weight coefficient λ is initialized as 0.8; z represents a distance offset between the current system state and the safety boundary of the barrier, z = d min - d safe , wherein d min is the minimum distance between the current mobile robot and the nearest barrier, d safe is a set safety distance threshold; and q is a weighted parameter of the state safety risk, used to describe the safety risk sensitivity at different proximity degrees. The input of the value network is the integrated state vector St and the original action a raw = (v raw , ω raw ), and the output is the estimated value of the state-action value Q(St, a) for evaluating the safety of the original action a raw = (v raw , ω raw ) and guiding the policy network to update; The safety action projection module projects the original action a raw = (v raw , ω raw ) constrained and projected into the absolute allowed safety action space of the mobile robot physical platform, to obtain the projected safety action a safe of the mobile robot, as follows: a safe =[cllip(v raw ,v min +δ,v max -δ),cllip(w raw ,w min +δ,w max -δ)] where clip is a clipping function, V min , V max and w min , W max are the speed, angular velocity bounds allowed by the hardware platform, and δ is a safety margin.

4. The method of claim 1, wherein: In step 3), the total reward function R total consists of a plurality of sub-reward items, including a target reward, a progress reward, a speed reward, a direction penalty, a safety distance, and a safety penalty. The total reward function R total is expressed by the formula: R total = w1R q + w2R c + w3R v + w4R d + w5R s + w6R b wherein: R q The reward for the mobile robot reaching the goal is set to the following function: wherein d p is the distance of the mobile robot from the target point, c p is a custom distance; R c For progress rewards, define as the following function: R c = (d previous -d current ) x k where d current is the Euclidean distance of the mobile robot to the target point at the current time step; d previous is the Euclidean distance of the mobile robot to the target point saved at the previous time step; An adaptive progress reward coefficient is introduced: wherein N fail is the number of recent obstacle avoidance failures; N total is the total number of training rounds; and β is an exploration weight. R v To reward speed, set to the following function: wherein d safe is a set safety distance threshold; d min is the minimum distance of the mobile robot to the nearest obstacle; λ is a self-defined coefficient, ranging from 0 to 0.8; the safety distance threshold d safe satisfies the following function: wherein d base is the basic safety distance, set to 0.6 m; v is the current linear speed of the mobile robot, v max is the maximum allowed linear speed; μ is a sensitivity coefficient, set to 0-0.5; R d For the direction penalty, set as the following function: R d = -a x |ω| wherein, a is a self-defined penalty coefficient, which is determined by experiment adjustment; ω is the angular velocity of the mobile robot; the penalty coefficient is designed as: g i penalty parameter; R s To penalize for safety distance, set to the following function: wherein, ∈ is a smoothing constant; an adaptive smoothing constant is introduced as follows: ∈ = 0.1 + γ × σ wherein, σ is the standard deviation of the laser radar distance measurement; γ is a noise sensitivity coefficient; R b For the safety penalty, which is designed based on an asymmetric barrier Lyapunov function V(z), is set as the following function: In the total reward function R total = w1R q + w2R c + w3R v + w4R d + w5R s + w6R b The weights of each sub-reward item are set to [w1, w2, w3, w4, w5, w6] = [1.0, 2.0, 0.5, 0.1, 0.8, 0.5].

5. The method of claim 1, wherein: In step 4), based on the integrated state vector St obtained in step 1) and the total reward function R obtained in step 3) total In the ROS / Gazebo simulation environment, the deep reinforcement learning network obtained in step 2) is iteratively trained and optimized in static and dynamic scenarios respectively. The method for obtaining the trained deep reinforcement learning network is: In a dynamic scenario, the set simulation environment includes randomly distributed static obstacles and multiple dynamic obstacles moving along preset paths; the initial pose of the mobile robot, the target point, and the motion trajectory of the dynamic obstacles are randomly generated; the training process is as follows: the mobile robot perceives the environment through the laser radar and obtains the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(v raw ,ω raw ); the original action a raw =(v raw ,ω raw ) is input into the value network and the safe action projection module at the same time; the value network uses the safe state-action value function Q safe to evaluate the safety of the action and guide the policy network to update; the safe action projection module projects the original action a raw =(v raw ,ω raw ) into the safe action space to obtain the safe action a safe ; the safe action a safe is used for environment interaction, and the reward is calculated based on the total reward function R total ; the mobile robot adjusts the strategy according to the reward value until it successfully avoids obstacles or reaches the target point; each round of training lasts at most 500 steps, and if collision occurs or the target point is reached, the training is terminated in advance; the mobile robot is counted as a successful simulation from the starting point to the target point without collision, and then the starting point and the target point are reset for a new round of simulation, and the training is repeated until the policy network converges; In the static scene, the set simulation environment includes a fixed map, fixed obstacles, a fixed starting point and a target point; the training process is as follows: the mobile robot perceives the environment through the laser radar, and obtains the integrated state vector St; the integrated state vector St is input into the policy network, and the policy network outputs the original action a raw =(v raw ,ω raw ); the original action a raw =(v raw ,ω raw ) is processed by the safety action projection module to obtain the safety action a safe ; the safety action a safe is used for environment interaction, and the reward is calculated based on the total reward function R total ; if collision occurs or the target point is reached, the starting point of the mobile robot is reset, but the current policy parameters are retained for continuous training; through the fixed scene training, the deep reinforcement learning network is strengthened to learn the basic obstacle avoidance strategy; Finally, a deeply trained and parameter optimized deep reinforcement learning network is obtained.

6. The method of claim 1, wherein: In step 5), the method of migrating the trained deep reinforcement learning network obtained in step 4) to a real mobile robot and performing navigation test experiments in static and dynamic environments respectively to realize a complete closed loop from simulation to real object is: An experimental scene is constructed in a static environment, and obstacles are laid out; a piece of open land is selected, and randomly distributed cylinders, cubes and diamond columns are set as static obstacles on the open land, and then a reference path is planned for the mobile robot, and a target point is set; The parameters of the deep reinforcement learning network are adjusted, and the weights of the sub-reward items in the total reward function are adjusted to verify the control response at different speeds; the parameters include the weights of the policy network, the entropy temperature coefficient, the weights of the value network, the safety weight coefficient λ, the weighted parameter q of the state safety risk in the asymmetric obstacle Lyapunov function, and the weights of the sub-reward items in the total reward function; The basic obstacle avoidance capability is verified in a controlled static environment, dynamic elements are gradually introduced, an experimental scene is constructed, dynamic obstacles are set, a random number of human-shaped and mobile robots are set to move randomly, the speed is set to 0-4 m / s, the mobile robot obtains point cloud data through a laser radar, predicts the dynamic obstacles, and generates a local map; hierarchical control debugging is performed through the policy network and the value network; when the mobile robot collides with the dynamic obstacle, a test is ended, human-robot interaction scene training data is counted, and the index of the mobile robot is evaluated; in the dynamic response aspect, the accuracy of trajectory prediction and the handling of sudden obstacles are evaluated, and the system performance is comprehensively evaluated.

Images