A mobile robot obstacle avoidance path planning method and system based on reinforcement learning

Abstract

This invention discloses a method and system for obstacle avoidance path planning of mobile robots based on reinforcement learning. The method includes: collecting mobile robot operating environment data and unifying the data collected by LiDAR, depth camera, and odometry into the robot's body coordinate system; constructing a local occupancy grid map containing static obstacle regions, dynamic obstacle candidate regions, unknown occlusion regions, and passable regions based on obstacle point cloud data and depth image data. This invention relates to the field of mobile robot autonomous navigation and path planning technology. This method and system for obstacle avoidance path planning of mobile robots, which constructs a local occupancy grid map composed of static obstacle regions, dynamic obstacle candidate regions, unknown occlusion regions, and passable regions, enables the robot to simultaneously perceive known obstacles, dynamic risks, occlusion risks, and passable spaces in the same spatial representation, improving the integrity of local spatial representation in complex environments.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and path planning for mobile robots, and more specifically, to a method and system for obstacle avoidance path planning for mobile robots based on reinforcement learning. Background Technology

[0002] Mobile robots, used in warehousing, indoor delivery, inspection, and public service scenarios, need to autonomously avoid obstacles in environments where static obstacles, pedestrians, mobile devices, and occluded areas coexist. Existing technologies include a patent (CN110632931A) that discloses a deep reinforcement learning-based collision avoidance planning method for mobile robots in dynamic environments. This method utilizes laser ranging data and neural networks to output robot actions, but it primarily relies on ranging input and lacks a unified expression for occlusion uncertainty, collision time, and passable corridors. A patent (CN114237235A) discloses a mobile robot obstacle avoidance method based on near-end policy optimization, which improves adaptability in multi-user environments, but it still focuses on policy output and lacks specific handling for state recognition and reward switching when the robot gets stuck in local optima. A patent (CN112882469A) discloses a deep reinforcement learning-based obstacle avoidance navigation method that uses global path waypoints as temporary targets, but its temporary targets mainly come from the initial global path and lack a mechanism for generating escape sub-targets in real time based on dynamic collision margin and occlusion risk. Therefore, existing technologies still have shortcomings in dynamic obstacle prediction, local trap identification, and autonomous escape. It is necessary to propose an obstacle avoidance path planning method and system that integrates multi-source perception, spatiotemporal risk representation, trap determination, and reward mode switching. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for obstacle avoidance path planning of mobile robots based on reinforcement learning. It solves the problems of insufficient recognition of the future movement trend of obstacles, occlusion uncertainty and local optimal trap state of existing mobile robots in dynamic environments, which leads to delayed obstacle avoidance response, path jitter and weak escape ability.

[0004] This invention achieves the above objective through the following technical solution: a mobile robot obstacle avoidance path planning method based on reinforcement learning, the method comprising: Acquire mobile robot operating environment data, and convert obstacle point cloud data collected by lidar, depth image data collected by depth camera, and robot pose, linear velocity and angular velocity collected by odometry into robot body coordinate system; A local occupancy grid map is constructed based on the obstacle point cloud data and depth image data. The local occupancy grid map consists of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas. Based on the local occupancy grid map and robot pose changes over multiple consecutive frames, the relative motion state of obstacles is estimated, and the robot's trajectory and obstacle occupancy area within the future time window are predicted for the candidate velocity command set, generating a spatiotemporal collision margin tensor. The spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance are input into the reinforcement learning policy network, which outputs linear velocity commands and angular velocity commands. Local optimal trap states are identified based on the target distance descent rate, number of course reversals, frequency of angular velocity sign switching, linear velocity change amplitude, and continuity of passable corridors within a sliding time window. When a local optimal trap state is identified, a temporary escape sub-target is determined according to the spatiotemporal collision margin tensor, and the reinforcement learning policy network is switched from the target approach reward mode to the trap escape reward mode. Linear velocity and angular velocity commands are converted into differential drive control commands, and closed-loop correction is performed in combination with drive wheel encoder feedback. The obstacle avoidance path planning results are also updated cyclically.

[0005] Furthermore, acquiring the mobile robot's operating environment data includes: Add acquisition timestamps to obstacle point cloud data, depth image data, and odometry data respectively; Based on the extrinsic parameters of the lidar, the extrinsic parameters of the depth camera, and the installation pose of the odometer, the transformation relationship between the coordinate systems of each sensor and the robot body coordinate system is established; Based on the acquisition time of the odometry data, the obstacle point cloud data and depth image data are time-aligned; When the deviation between any data frame and the reference acquisition time is greater than the synchronization threshold, the data frame is discarded or interpolated for compensation. The time-aligned data is unified into the robot's body coordinate system according to the aforementioned transformation relationship, forming an environmental state input at the same moment.

[0006] Furthermore, the construction of the local occupancy raster map includes: Outlier removal and height filtering are performed on obstacle point cloud data to retain point clouds within the robot's collision range; The planned area around the robot is divided into grid cells according to the grid resolution, and the static obstacle area is determined based on the number of point clouds falling into the grid cells and the occupancy probability. Dynamic obstacle candidate regions are determined based on the changes in the occupancy position of the same grid neighborhood in consecutive frames; The unknown occlusion area is determined by the location of the depth abrupt change in the depth image data corresponding to the blind zone, low reflection area or occlusion boundary of the lidar. Grid cells that are not occupied by static obstacle areas, dynamic obstacle candidate areas, or unknown occlusion areas, and whose distance from the obstacle boundary is not less than the robot's outer outline safety distance, are defined as passable areas.

[0007] Furthermore, the generation of the spatiotemporal collision margin tensor includes: Based on the position difference of the centroid of the obstacle region in multiple consecutive frames of local occupancy grid map and the robot pose increment, calculate the direction and speed of the obstacle relative to the robot. A set of candidate speed commands is generated based on the robot's maximum linear velocity, maximum angular velocity, maximum acceleration, and minimum turning radius; For each candidate velocity command, calculate the robot's candidate motion trajectory within a future time window according to the discrete predicted step size; Based on the relative direction and speed of the obstacle's movement, generate the predicted area to be occupied by the obstacle within the future time window; Calculate the minimum safe distance between each candidate trajectory and the predicted obstacle occupancy area, the estimated collision time, the dynamic occupancy change, the occupancy uncertainty, and the corridor continuity value; The above calculation results are organized according to the candidate action dimension, time dimension, and space dimension to form a spatiotemporal collision margin tensor consisting of a static occupancy layer, a dynamic residual layer, a collision time layer, an occlusion uncertainty layer, and a passable corridor layer.

[0008] Furthermore, the reinforcement learning policy network is trained using a proximal policy optimization algorithm, and the training process includes: The state space is defined by the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance. Linear velocity commands and angular velocity commands serve as the continuous motion space; In the target approach reward mode, a reward signal is constructed based on the reduction in target distance, the maintenance of safe distance, the change in path length, the smoothness of motion, and the satisfaction of kinematic constraints. In the trap escape reward mode, a reward signal is constructed based on the approach of the temporary escape sub-target, the distance from the closed obstacle boundary, the increase in the width of the passable corridor, the decrease in the number of course reversals, and the reduction in the risk of approaching dynamic obstacles. The policy network parameters are updated by using a proximal policy optimization algorithm to ensure that the output linear velocity and angular velocity commands satisfy the robot's kinematic constraints.

[0009] Furthermore, the identification of locally optimal trap states includes: Calculate the target distance descent rate between the robot and the final target within a sliding time window; The number of times the robot's heading angle change direction reverses within the sliding time window is counted to obtain the heading reversal count; The frequency of angular velocity sign switching is obtained by counting the number of times the sign of the angular velocity changes from positive to negative or from negative to positive within the sliding time window. The difference between the maximum and minimum values ​​of the linear velocity within the sliding time window is calculated to obtain the amplitude of the linear velocity change. The continuity of the accessible corridors is obtained by considering the changes in the connectivity length and direction of the main accessible corridors in the accessible corridor layer. When the target distance descent rate is less than the descent rate threshold, and any one of the following indicators—the number of course reversals, the frequency of angular velocity sign switching, or the continuity of the passable corridor—satisfies the trap threshold condition, the robot is determined to be in a local optimal trap state.

[0010] Furthermore, determining the temporary escape sub-target includes: Extract candidate travel directions that connect to the robot's current position from the accessible corridor layer; Candidate travel directions with expected collision times less than a safe time threshold are eliminated based on the collision time layer; Candidate travel directions with occlusion uncertainty values ​​greater than the uncertainty threshold are eliminated based on the occlusion uncertainty layer; For the retained candidate travel directions, calculate the corridor width, continuous length, angle with the final target direction, and dynamic obstacle approach speed; Based on the sorting rules of the corridor width, continuous length, angle with the final target direction, and approach speed of dynamic obstacles, the candidate passage direction with the highest priority for getting out of trouble is determined; In the candidate passage direction with the highest escape priority, grid points that are not less than the robot's outer perimeter safety distance from the obstacle boundary and are within the reachable range of the future time window are selected as temporary escape sub-targets.

[0011] Furthermore, the reward mode switching includes: When no local optimal trap state is identified, the target-approaching reward mode is maintained, so that the reinforcement learning policy network prioritizes outputting actions that are toward the final target and satisfy the safe distance constraint. After identifying a local optimal trap state and generating a temporary escape sub-target, the trap escape reward mode is activated, causing the reinforcement learning policy network to prioritize outputting actions that approach the temporary escape sub-target, move away from the closed obstacle boundary, and reduce the course reversal. When the distance between the robot and the temporary escape sub-target is less than the escape distance threshold, the target distance decrease rate is greater than the recovery threshold, and the continuity of the passable corridor meets the recovery conditions, the temporary escape sub-target is deleted, and the reinforcement learning policy network is restored to the target approach reward mode.

[0012] Furthermore, the differential drive closed-loop control includes: Calculate the target rotational speeds of the left and right drive wheels based on the linear velocity command, angular velocity command, distance between the left and right drive wheels, and drive wheel radius. Read the actual rotational speeds fed back by the encoders of the left and right drive wheels, and calculate the speed deviations of the left and right drive wheels respectively; The control values ​​of the left and right drive wheel motors are adjusted based on the speed deviations of the left and right drive wheels. The mobile robot is driven to move according to the corrected control values ​​of the left and right drive wheel motors; The robot's pose, velocity, and environmental perception data after movement are fed back into the local occupancy grid map update process, and the spatiotemporal collision margin tensor and obstacle avoidance actions are regenerated until the robot reaches the final goal or receives a stop command.

[0013] A reinforcement learning-based obstacle avoidance path planning system for mobile robots, used to execute the aforementioned reinforcement learning-based obstacle avoidance path planning method for mobile robots, the system comprising: The environmental data acquisition module is used to collect obstacle point cloud data, depth image data, robot pose, linear velocity and angular velocity, and unify the collected data into the robot body coordinate system. The local map building module is used to generate a local occupancy raster map consisting of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas based on obstacle point cloud data and depth image data. The collision margin generation module is used to generate a spatiotemporal collision margin tensor based on multiple consecutive frames of local occupancy grid map, robot pose changes, and candidate velocity command set. The reinforcement learning decision module is used to input the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance into the reinforcement learning policy network, and output linear velocity commands and angular velocity commands. The trap identification module is used to identify the local optimal trap state based on the target distance descent rate, the number of course reversals, the frequency of angular velocity sign switching, the magnitude of linear velocity change, and the continuity of the passable corridor within the sliding time window. The escape sub-target generation module is used to determine temporary escape sub-targets based on the passable corridor layer, collision time layer, and occlusion uncertainty layer; The reward mode switching module is used to switch the reinforcement learning policy network between the goal approach reward mode and the trap escape reward mode; The differential closed-loop control module is used to convert linear velocity commands and angular velocity commands into control quantities for the left and right drive wheels, and to perform closed-loop correction based on feedback from the drive wheel encoders.

[0014] The beneficial effects of this invention are as follows: 1. By unifying LiDAR point cloud, depth image, and odometry data into the robot's body coordinate system, and constructing a local occupancy grid map consisting of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas, the robot can simultaneously perceive known obstacles, dynamic risks, occlusion risks, and passable spaces in the same spatial representation, thereby improving the integrity of local spatial representation in complex environments.

[0015] 2. By estimating the relative motion state of obstacles through continuous multi-frame local occupancy grid maps and robot pose changes, a spatiotemporal collision margin tensor containing a static occupancy layer, a dynamic residual layer, a collision time layer, an occlusion uncertainty layer, and a passable corridor layer is generated. This enables the reinforcement learning policy network to make obstacle avoidance decisions based on future collision time, occlusion uncertainty, and passable corridor continuity, thereby improving the foresight and safety of dynamic obstacle avoidance.

[0016] 3. By monitoring the target distance descent rate, the number of heading reversals, the frequency of angular velocity sign switching, the magnitude of linear velocity changes, and the continuity of passable corridors through a sliding time window, it can effectively distinguish between normal deceleration and avoidance and local optimal trap states, reducing the probability of the robot swaying in place, repeatedly turning, and being unable to escape for a long time in narrow passages, concave obstacle areas, or areas with frequent interference from dynamic obstacles.

[0017] 4. After identifying the local optimal trap state, a temporary escape sub-target is generated based on the passable corridor layer, collision time layer, and occlusion uncertainty layer. The reinforcement learning policy network is switched from the target approach reward mode to the trap escape reward mode, so that the robot prioritizes actions such as moving away from the closed obstacle boundary, increasing the width of the passable corridor, and approaching the temporary escape sub-target, thereby improving the mobile robot's autonomous escape capability in complex local environments.

[0018] 5. The linear velocity and angular velocity commands output by the reinforcement learning policy network are converted into control quantities for the left and right drive wheels, and closed-loop correction is performed in combination with the feedback from the drive wheel encoder. This enables the output of the planning layer to be stably mapped to the execution layer, improving the continuity, smoothness and actual executability of the obstacle avoidance actions of the mobile robot. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a schematic diagram of the local occupancy grid map construction of the present invention; Figure 3 This is a schematic diagram illustrating the generation of the spatiotemporal collision margin tensor in this invention; Figure 4 This is a flowchart of the local optimal trap identification and temporary escape sub-target generation process of the present invention; Figure 5 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0021] Example 1: Please see Figure 1-5 This invention provides a reinforcement learning-based obstacle avoidance path planning method for mobile robots, applied to differential-drive mobile robots. The mobile robot includes a LiDAR, depth camera, odometer, left and right drive wheel encoders, a computing unit, and a motion controller. The computing unit performs tasks such as fusion of operating environment data, construction of a local occupancy grid map, generation of a spatiotemporal collision margin tensor, output of reinforcement learning policy network actions, identification of local optimal trap states, generation of temporary escape sub-targets, and switching of reward modes. The motion controller converts the linear velocity and angular velocity commands output by the reinforcement learning policy network into control quantities for the left and right drive wheels and performs closed-loop correction based on feedback from the left and right drive wheel encoders.

[0022] In this embodiment, the circumscribed circle radius of the mobile robot is 0.22 m to 0.35 m, the maximum linear velocity is 0.6 m / s to 1.2 m / s, the maximum angular velocity is 1.0 rad / s to 2.5 rad / s, and the control cycle is 0.05 s to 0.10 s. These parameters are determined based on the robot chassis dimensions, motor output capacity, braking distance, and indoor passage width. The control cycle should be less than one-tenth of the robot's minimum braking time, and the robot's displacement within one control cycle should not exceed twice the side length of a single grid cell in the locally occupied grid map, thus ensuring timely updates to obstacle avoidance actions.

[0023] The lidar is used to collect point cloud data of obstacles around the robot, with a scanning frequency of 10 Hz to 20 Hz; the depth camera is used to collect depth image data in front of the robot, with a sampling frequency of 15 Hz to 30 Hz; the odometry is used to collect the robot's current pose, linear velocity, and angular velocity, with a sampling frequency no lower than the motion control frequency. The rule for determining the sensor sampling frequency is: the sensor sampling period should be less than half the time required for the robot to pass through a local grid at its maximum linear velocity, in order to reduce the probability of missed detection when dynamic obstacles rapidly enter in front of the robot.

[0024] When the mobile robot is running, the computing unit first acquires environmental data. Specifically, it collects point cloud data of obstacles around the robot using a LiDAR, depth image data of the robot's front using a depth camera, and the robot's current pose, linear velocity, and angular velocity using an odometry system. The computing unit adds a timestamp to the obstacle point cloud data, depth image data, and odometry data, and synchronizes the obstacle point cloud data and depth image data using the odometry data acquisition time as a reference. When the deviation between any data frame and the reference acquisition time exceeds a synchronization threshold, the data frame is discarded or interpolated for compensation. The synchronization threshold is set between 0.03 s and 0.08 s, preferably 0.05 s; it is determined based on the following: the synchronization threshold is not greater than half the time required for the robot to pass through a local grid at its maximum linear velocity, and not greater than twice the sampling period of the depth camera.

[0025] After time synchronization is completed, the computational processing unit converts the data collected by each sensor to the robot's body coordinate system based on the extrinsic parameters of the lidar, the extrinsic parameters of the depth camera, and the installation pose of the odometer. The coordinate transformation process satisfies the following formula (1):

[0026] in, Represents the spatial coordinates of a point in the sensor coordinate system, in units of ; Represents the spatial coordinates of a point in the robot's body coordinate system, in units of 1. ; Represents the rotation matrix from the sensor coordinate system to the robot body coordinate system; it is dimensionless. This represents the translation vector from the origin of the sensor coordinate system to the origin of the robot body coordinate system, in units of... ; Subscript Indicates the coordinate system of the lidar or depth camera, subscript This represents the robot's body coordinate system. The rotation matrix and translation vector are determined through sensor extrinsic parameter calibration. During calibration, the center of the robot chassis is taken as the origin of the robot's body coordinate system, and the robot's forward direction is taken as the positive direction of the robot's body coordinate system. If the spatial position deviation of the same stationary calibration point after transformation to the robot's body coordinate system by different sensors is greater than 0.03 m, the extrinsic parameter calibration is performed again to ensure that multi-source sensing data participates in subsequent calculations under the same coordinate reference.

[0027] After obtaining the operating environment data in a unified coordinate system, the computing and processing unit constructs a local occupancy grid map. The local occupancy grid map is constructed with the robot's current position as the center, with a local planning range of 6 to 10 meters in the front-back direction and 4 to 8 meters in the left-right direction; the grid resolution is 0.05 to 0.10 meters. The rules for determining the grid resolution are: firstly, the grid resolution should not exceed one-third of the robot's circumcircle radius to ensure the accuracy of obstacle boundary representation; secondly, the grid resolution should be greater than three times the random error of the lidar ranging to reduce grid occupancy jumps caused by single-frame noise. For indoor delivery robots, a grid resolution of 0.05 meters is preferred; for warehouse handling robots, a grid resolution of 0.10 meters is preferred.

[0028] The computational processing unit performs outlier removal and height filtering on the obstacle point cloud data, retaining point clouds within the robot's collision-allowed height range. The collision-allowed height range is determined based on the robot's chassis height, sensor mounting height, and actual obstacle contact height, ranging from 0.05 m to 1.20 m. If the average distance between a point cloud and its neighboring points is greater than three times the average distance of the neighboring points, the point cloud is identified as an outlier and removed. Subsequently, the retained valid point clouds are projected onto a local occupancy grid map, and static obstacle areas are determined based on the number of grid cells the point clouds fall into and their occupancy probability.

[0029] For dynamic obstacle candidate regions, the computational processing unit identifies them based on the positional changes of obstacle regions within the same grid neighborhood in a multi-frame local occupancy grid map. When the centroid of the same obstacle region undergoes continuous displacement within 3 to 5 consecutive frames, and the displacement is greater than one grid resolution, the obstacle region is marked as a dynamic obstacle candidate region. For LiDAR blind spots, low-reflection areas, and occlusion boundaries, the computational processing unit uses depth image data for supplementary judgment. When the depth abrupt change of adjacent pixel regions in the depth image is greater than 0.30 m, and the region is not effectively covered by LiDAR, the corresponding grid is marked as an unknown occlusion region. Grid cells that are not occupied by static obstacle regions, dynamic obstacle candidate regions, or unknown occlusion regions, and whose distance from the obstacle boundary is not less than the robot's outer perimeter safety distance, are determined to be passable areas.

[0030] The robot's outer perimeter safety distance is set between 0.25 m and 0.60 m, determined based on the robot's circumcircle radius, braking distance at maximum linear velocity, sensor ranging error, and displacement margin corresponding to control delay. For densely populated environments, the robot's outer perimeter safety distance is taken as the larger value within this range; for enclosed warehouse passageways, the robot's outer perimeter safety distance is taken as no less than the robot's circumcircle radius plus 0.10 m. Therefore, the local occupancy grid map includes static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas, thus simultaneously representing known obstacles, dynamic risks, occlusion risks, and local passable space.

[0031] Furthermore, the computational processing unit generates a spatiotemporal collision margin tensor based on continuous multi-frame local occupancy grid maps and robot pose changes. Specifically, for dynamic obstacle candidate regions, the region centroid is extracted in consecutive frames, and combined with the robot's own pose increment, the motion direction and velocity of the obstacle relative to the robot's body coordinate system are estimated. The effective range of the obstacle's relative velocity is 0 m / s to 2.0 m / s; if the estimated velocity exceeds 2.0 m / s and the duration is less than 0.3 s, it is considered that the estimation is caused by occlusion jumps or point cloud mismatches, and it is corrected using adjacent stable frames.

[0032] Subsequently, the computational processing unit generates a set of candidate velocity commands based on the robot's maximum linear velocity, maximum angular velocity, maximum acceleration, and minimum turning radius. The candidate velocity commands include candidate linear velocities and candidate angular velocities. Candidate linear velocities are discretized from 0 m / s to the maximum linear velocity, with discretization intervals ranging from 0.05 m / s to 0.10 m / s; candidate angular velocities are discretized from the negative maximum angular velocity to the positive maximum angular velocity, with discretization intervals ranging from 0.10 rad / s to 0.25 rad / s. Candidate velocity commands that cannot satisfy the robot's acceleration constraints or minimum turning radius constraints within one control cycle are not included in subsequent trajectory prediction.

[0033] For each candidate velocity command, the computational processing unit predicts the robot's candidate trajectory within a future time window. The future time window ranges from 2.0 s to 4.0 s, and the prediction step size ranges from 0.10 s to 0.20 s. The future time window is determined by the following rules: it must be no less than the sum of the time required for the robot to decelerate from its maximum linear velocity to a stop and the perception and control delay, and no greater than the shortest travel time that can be covered by the local occupied grid map area, thus avoiding long-term predictions exceeding the sensor's reliable area. The prediction step size is determined by the following rule: the robot's displacement within a single prediction step must not exceed twice the resolution of one grid cell.

[0034] While predicting candidate robot trajectories, the computational processing unit predicts the area occupied by obstacles within a future time window based on the relative motion direction and speed of the obstacles. For each candidate speed command, the estimated collision time and minimum space margin between the candidate trajectory and the predicted area occupied by the obstacle are calculated, as shown in formula (2):

[0035]

[0036] in, Indicates the candidate speed command number; This indicates the predicted time within a future time window, in units of... ; Indicates the length of the future time window, in units of ; Indicates candidate speed command The robot at the predicted time The trajectory position, in units of ; Indicates the obstacle at the predicted time. The predicted occupied area; Indicates candidate speed command The shortest distance between the robot's trajectory position and the predicted area occupied by obstacles, in units of ; This indicates the safe distance for the robot's outer contour, in units of... ; Indicates candidate speed command The corresponding estimated collision time, in units of ; Indicates candidate speed command The corresponding minimum space margin, in units of If no predicted moment that meets the collision conditions exists within the future time window, then... Set as This indicates that the candidate velocity command did not collide within the prediction time window. Less than If the candidate motion trajectory enters the safe distance range of the robot's outline, the candidate speed command is judged as a high-risk command.

[0037] The robot's outer perimeter safety distance ds in the above formula is determined based on the robot's circumcircle radius, maximum linear velocity braking distance, sensor ranging error, and control delay displacement margin. The expected collision time safety threshold is taken as 0.8 s to 1.5 s, determined by the rule that it should not be less than the sum of the robot's emergency braking time, perception processing delay, and control execution delay, with a safety margin of 0.2 s to 0.5 s reserved. The spatiotemporal collision margin tensor generated from the above calculation results is organized according to the candidate action dimension, time dimension, and spatial dimension, and includes at least a static occupancy layer, a dynamic residual layer, a collision time layer, an occlusion uncertainty layer, and a passable corridor layer.

[0038] The system comprises the following layers: a static occupancy layer, representing the spatial constraints of fixed obstacles on candidate trajectories; a dynamic residual layer, representing the degree of change in obstacle regions between consecutive frames; a collision time layer, representing the expected collision time corresponding to the candidate trajectories; an occlusion uncertainty layer, representing the degree of risk uncertainty related to LiDAR blind spots, abrupt depth changes, or occlusion boundaries; and a walkable corridor layer, representing the connection direction, corridor width, and continuous length between the robot's current position and the walkable area. Through this tensor representation, the reinforcement learning policy network can simultaneously obtain static risk, dynamic risk, collision time margin, occlusion uncertainty, and walkable corridor information before outputting actions.

[0039] Subsequently, the computational processing unit constructs the reinforcement learning state input from the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target distance, and inputs this input into the reinforcement learning policy network. The reinforcement learning policy network is trained using a proximal policy optimization algorithm. Its state space includes the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target distance. Its continuous action space includes linear velocity commands and angular velocity commands. The unit for linear velocity commands is m / s, and the unit for angular velocity commands is rad / s.

[0040] In the target approach reward mode, the reward signals include the reduction in target distance, the maintenance of safe distance, the change in path length, motion smoothness, and the satisfaction of kinematic constraints. All of these reward items are normalized before training to map different physical quantities to a dimensionless range. To avoid the problem of weight mismatch in different scenarios due to fixed weight addition, the reward items in this embodiment are enabled sequentially according to the training stage: in the early stage of training, safe distance and kinematic constraints are enabled, allowing the policy network to prioritize learning collision avoidance and avoiding exceeding limits; in the middle stage of training, target approach and path length constraints are enabled, allowing the policy network to learn task efficiency; in the later stage of training, motion smoothness constraints are enabled, ensuring continuous and executable output actions. If the robot collides or enters the safe distance range, the training round is penalized, causing the reinforcement learning policy network to prioritize forming a safe obstacle avoidance strategy.

[0041] During the robot's obstacle avoidance maneuvers, the computational processing unit simultaneously identifies locally optimal trap states. Specifically, within a sliding time window, it calculates the target distance descent rate between the robot and the final target, counts the number of times the robot's heading angle changes in the opposite direction (resulting in the number of heading reversals), counts the number of times the angular velocity sign changes from positive to negative or vice versa (resulting in the angular velocity sign switching frequency), calculates the difference between the maximum and minimum linear velocity values ​​within the sliding time window (resulting in the linear velocity change amplitude), and determines the continuity of the passable corridors based on the connectivity length and direction changes of the main passable corridors in the passable corridor layer. The sliding time window is set to 3 to 6 seconds, determined by the following rules: it must be no less than the time required for the robot to complete one deceleration, turning, and re-acceleration, and no more than 20 times the effective update cycle of the locally occupied grid map.

[0042] The local optimal trap state is determined by formula (3):

[0043] in, This indicates a locally optimal trap state flag, with a value of [value missing]. This indicates that the robot has entered a local optimum trap state, with a value of This indicates that the robot has not entered a local optimum trap state; This represents the rate of decrease in target distance within the sliding time window, in units of ; This represents the target distance descent rate threshold, in units of... ; This indicates the number of heading reversals within the sliding time window, expressed in times. This represents the threshold number of course reversals, expressed in times. Indicates the frequency of angular velocity sign switching, in units of ; This represents the angular velocity sign switching frequency threshold, in units of... ; Indicates the continuity of a passable corridor, dimensionless, with a value range of [value missing]. to ; This represents the continuity threshold of a passable corridor and is dimensionless.

[0044] The rule for determining the above threshold is: target distance descent rate threshold. Take the target distance decrease rate from the historical normal obstacle avoidance sample. The value corresponding to the percentile, and not lower than ; Threshold for the number of times the course is reversed Pick Next This threshold is then adjusted based on the robot's minimum turning radius; the smaller the minimum turning radius, the larger the threshold value. (Angular velocity sign switching frequency threshold) Pick to And less than one-fifth of the control frequency; passable corridor continuity threshold Pick to The determination is based on the connectivity length and directional stability statistics of passable grids in the local occupied grid map. The robot is only considered to have entered a local optimal trap state when at least one of the following phenomena occurs simultaneously: insufficient target distance reduction, repeated heading, frequent switching of angular velocity, or instability of passable corridors. This reduces the probability of misjudging the obstacle as a trap during normal obstacle avoidance.

[0045] When the computational processing unit determines that the robot has entered a local optimum trap state, it generates a temporary escape sub-target based on the spatiotemporal collision margin tensor. Specifically, it extracts candidate travel directions connected to the robot's current position from the passable corridor layer. The angular interval of the candidate travel directions is taken as... to The values ​​are then converted to radians for calculation. Next, candidate travel directions with predicted collision times less than a safe time threshold are eliminated based on the collision time layer; candidate travel directions with occlusion uncertainty values ​​greater than an uncertainty threshold are eliminated based on the occlusion uncertainty layer. The uncertainty threshold is set to... to The determination is based on the effective pixel ratio of the depth image, the coverage of the lidar, and the intensity of the depth change at the occlusion boundary. In densely populated environments, the uncertainty threshold is set to a lower value; in static warehousing environments, the uncertainty threshold is set to a higher value.

[0046] For the retained candidate passage directions, the computational processing unit extracts the corresponding corridor width, continuous length, angle with the final target direction, and dynamic obstacle approach speed, and uses an unweighted lexicographical order rule to determine the temporary escape sub-target, as shown in formula (4):

[0047]

[0048] in, Indicates candidate escape points; Represents the set of candidate escape points; This indicates the final selected temporary escape sub-target; This indicates that the selection is maximized according to lexicographical order. This represents the width of the passageway corresponding to the candidate escape point, in units of... ; This represents the continuous length of the passageway corresponding to the candidate escape point, in units of... ; This represents the angle between the direction of the candidate escape point and the direction of the final target, in units of... ; This represents the approach velocity of a dynamic obstacle along the direction of the candidate escape point, in units of... ; Indicates the estimated collision time in the direction of the candidate escape point, in units of ; This represents the safe time threshold, in units of... ; This represents the occlusion uncertainty in the direction of the candidate escape point, and is dimensionless. This represents an uncertain threshold for occlusion and is dimensionless. This represents the distance between the candidate escape point and the nearest obstacle boundary, in units of... ; This indicates the safe distance for the robot's outer contour, in units of... .

[0049] The above lexicographical order rule means: Under the premise of satisfying the constraints of expected collision time, occlusion uncertainty, and obstacle boundary safety distance, the candidate escape point with the largest passageway width is selected first; if the difference in passageway width is less than one grid resolution, the continuous length of the passageway is compared; if the difference in continuous length of the passageway is less than two grid resolutions, the angle between the candidate escape point's direction and the final target direction is compared; if the angle difference is less than... Then, the approach speed of dynamic obstacles is compared. This unweighted selection method avoids the problem of parameter mismatch in different environments caused by adding fixed weights, thus giving the generation process of temporary escape sub-targets clear priority and interpretability.

[0050] After generating the temporary escape sub-target, the computational processing unit switches the reinforcement learning policy network from the target approach reward mode to the trap escape reward mode. In the trap escape reward mode, reward signals include the approach amount of the temporary escape sub-target, the distance from the closed obstacle boundary, the increase in the width of the passable corridor, the reduction in the number of course reversals, and the reduction in the risk of approaching dynamic obstacles. These reward items are also first dimensionlessly normalized and then enabled sequentially according to the training order of safety, escape, and smoothness. Specifically, in the early stages of training, priority is given to constraining the robot to avoid entering collision zones and high uncertainty zones; in the middle stages of training, the robot's approach to the temporary escape sub-target is strengthened and the width of the passable corridor is increased; in the later stages of training, motion smoothness constraints are added to reduce sudden stops and sharp turns during the escape process.

[0051] The robot is considered to have successfully escaped when the distance between it and the temporary escape target is less than the escape distance threshold, the target distance decrease rate is greater than the recovery threshold, and the continuity of the passable corridor meets the recovery conditions. The escape distance threshold is set as follows: to The determination is based on the robot's circumcircle radius and grid resolution; the recovery threshold is taken as the th value of the target distance decrease rate in normal obstacle avoidance samples. The percentile value; the condition for restoring the continuity of a passable corridor is continuity. to The main internal passageway remains connected, and the direction of the main passageway changes by less than [a certain amount]. After successfully escaping the obstacle, the computational processing unit removes the temporary escape sub-objective and restores the reinforcement learning policy network to the target-approaching reward mode, enabling the robot to continue executing obstacle avoidance path planning toward the final goal.

[0052] Finally, the computational processing unit sends the linear velocity and angular velocity commands output by the reinforcement learning policy network to the motion controller. The motion controller calculates the target rotational speeds of the left and right drive wheels based on the linear velocity commands, angular velocity commands, the distance between the left and right drive wheels, and the drive wheel radii. The distance between the left and right drive wheels is taken as... to The radius of the drive wheel is taken as to The specific values ​​are determined by the chassis structure of the mobile robot. The motion controller reads the actual rotational speeds fed back by the encoders of the left and right drive wheels, and calculates the speed deviations of the left and right drive wheels respectively. Then, it corrects the control quantities of the left and right drive wheel motors based on the speed deviations, so that the actual wheel speeds gradually approach the target wheel speeds.

[0053] During closed-loop control, the encoder speed feedback frequency is not lower than the control frequency. Times. If consecutive The actual wheel speed deviation exceeds the target wheel speed within each control cycle. If so, reduce the current linear velocity command and keep the angular velocity command below the angular velocity of the previous control cycle. To prevent the robot from slipping, making sharp turns, or colliding near low-attachment surfaces or obstacle boundaries, the mobile robot feeds back its new pose, velocity, point cloud data, and depth image data to the local occupancy grid map update process after performing an action, and regenerates the spatiotemporal collision margin tensor and obstacle avoidance actions. This process is repeated until the robot reaches the final target or receives a stop command.

[0054] Example 2: Based on the same inventive concept, this embodiment provides a mobile robot obstacle avoidance path planning system based on reinforcement learning. The system includes an environmental data acquisition module, a local map construction module, a collision margin generation module, a reinforcement learning decision-making module, a trap identification module, an escape sub-target generation module, a reward mode switching module, and a differential closed-loop control module.

[0055] The environmental data acquisition module is used to collect obstacle point cloud data, depth image data, robot pose, linear velocity, and angular velocity, and unify the collected data into the robot body coordinate system according to formula (1). The local map construction module is used to generate a local occupancy grid map composed of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas based on obstacle point cloud data and depth image data. The collision margin generation module is used to generate a spatiotemporal collision margin tensor based on multiple consecutive frames of local occupancy grid maps, robot pose changes, and candidate velocity command sets, and calculate the expected collision time and minimum spatial margin of candidate motion trajectories according to formula (2). The reinforcement learning decision module is used to input the spatiotemporal collision margin tensor, the robot's current velocity, the target relative direction, and the target distance into the reinforcement learning policy network, and output linear velocity commands and angular velocity commands.

[0056] The trap identification module identifies the local optimal trap state according to formula (3) based on the target distance descent rate, heading reversal times, angular velocity sign switching frequency, linear velocity change amplitude, and traversable corridor continuity within the sliding time window. The escape sub-target generation module determines the temporary escape sub-target according to formula (4) based on the traversable corridor layer, collision time layer, and occlusion uncertainty layer. The reward mode switching module switches the reinforcement learning policy network between the target approach reward mode and the trap escape reward mode. The differential closed-loop control module converts the linear velocity command and angular velocity command into control quantities for the left and right drive wheels and performs closed-loop correction based on the encoder feedback of the left and right drive wheels.

[0057] Example 3: This embodiment uses an autonomous delivery scenario of a mobile robot within a warehouse aisle as an example to further illustrate the reinforcement learning-based obstacle avoidance path planning method for mobile robots of the present invention. The mobile robot in this embodiment is a differential-drive transport robot with an external circumscribed circle radius of [missing information]. The distance between the left and right drive wheels is The radius of the drive wheel is The maximum linear velocity is The maximum angular velocity is The control cycle is The robot is equipped with a LiDAR, a forward depth camera, an odometer, and encoders for the left and right drive wheels. The LiDAR scanning frequency is [missing information]. The depth camera sampling frequency is The odometer sampling frequency is The encoder feedback frequency is .

[0058] In this embodiment, the work area is the warehouse rack aisle, and the aisle width is [missing information]. The robot needs to move from the starting point to a distance from the starting point. The target storage location. Fixed shelving is located on the left and right sides of the aisle, and there are [unclear - possibly referring to storage locations] within the aisle. Pedestrians crossing sideways and A temporary parking transport vehicle. Fixed shelves constitute static obstacles, pedestrians constitute dynamic obstacles, and the area behind the transport vehicle constitutes an unknown occlusion area. To ensure the robot has sufficient obstacle avoidance margin in narrow passages, this embodiment sets the front-to-back range of the partially occupied grid map to [missing information]. The left and right directional range is set to The raster resolution is set to The grid resolution is less than one-third of the robot's circumcircle radius and greater than the random error of the lidar ranging. This allows for a balance between the accuracy of obstacle boundary representation and the stability of the raster state.

[0059] Robot outline safety distance set to This value is determined by the radius of the robot's circumcircle. Braking distance at maximum linear velocity Sensor ranging error and the displacement margin corresponding to the control delay This was determined jointly. Because pedestrians cross the warehouse aisles in this embodiment, the safety distance was not calculated using the robot's circumscribed circle radius plus... Instead of using the smaller value, we adopt To improve obstacle avoidance safety in human-machine mixed scenarios.

[0060] After the robot starts, the LiDAR collects point cloud data of surrounding obstacles, the depth camera collects depth image data of the front, and the odometry collects the robot's current pose, linear velocity, and angular velocity. The computing unit uses the odometry timestamp as a reference to synchronize the point cloud data and depth image data. In this embodiment, the synchronization threshold is set to... When the deviation between the point cloud frame or depth image frame and the odometry reference time is greater than At that time, the computing and processing unit performs interpolation compensation or culling on the data frame. After time synchronization is completed, the computing and processing unit converts the lidar point cloud and depth camera point cloud to the robot body coordinate system according to formula (1).

[0061] During the localized raster map construction phase, the computational processing unit performs height filtering on the point cloud data, retaining only points with heights within a certain range. to Point cloud data within the specified range. For continuous and stable point cloud regions corresponding to the edge of the shelf, the computational processing unit marks them as static obstacle regions; for pedestrians in continuous... The centroid position change in the local occupancy of the frame in the raster map, if the centroid displacement exceeds If the area is obstructed by the lidar and the depth image shows a significant abrupt change, the processing unit marks it as an unknown obstructed area. Other unoccupied areas with a distance of at least [missing information] from the obstacle boundary are also considered. The grid cells are marked as passable areas.

[0062] During the spatiotemporal collision margin tensor generation stage, the computational processing unit estimates the relative motion velocity of the pedestrian based on multiple consecutive frames of local occupancy grid maps. In this embodiment, the lateral approach velocity of the pedestrian relative to the robot is... The direction is from the robot's right front towards the left-hand shelf. The computing unit sets the future time window to... The predicted step size is The predicted time window is larger than the sum of the time required for the robot to decelerate from its maximum linear velocity to a stop and the perception and control delay, but it does not exceed the reliable coverage range of the local map. Candidate linear velocities are determined according to... to Discretize, with a discrete interval of . Candidate angular velocities according to to Discretize, with a discrete interval of . Candidate speed commands that do not meet the maximum acceleration constraint or minimum turning radius constraint are directly rejected by the calculation and processing unit.

[0063] The computational processing unit calculates the estimated collision time and minimum space margin corresponding to each candidate velocity command according to formula (2). When the robot is in front When a pedestrian is detected crossing laterally, the estimated collision time for some candidate speed commands is less than [time value missing]. And the minimum space margin is less than These candidate velocity commands are deemed high-risk and will not be used as priority actions for the reinforcement learning policy network. The expected collision time safety threshold is set to... This threshold is determined by the robot's emergency braking time. Sensing processing delay Control execution delay and safety margin Together, they are determined. Through the above processing, the collision time layer in the spatiotemporal collision margin tensor can reflect the short-term collision risk caused by pedestrians crossing laterally, the occlusion uncertainty layer can reflect the potential risk of the invisible area behind the turnover vehicle, and the passable corridor layer can reflect the continuity of the passable space in front of the robot's left and right sides.

[0064] During the action output phase of the reinforcement learning policy network, the computational processing unit inputs the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target distance into the reinforcement learning policy network. Initially, the robot is in target-approaching reward mode, and the policy network outputs linear velocity commands. Angular velocity command The robot moves slightly to the left of the centerline of the passage. When a pedestrian enters the predicted area in front of the robot, the policy network reduces the linear velocity based on the collision time layer and the dynamic residual layer, and outputs a smaller angular velocity, causing the robot to decelerate and avoid the pedestrian in the direction with the wider passable corridor. Because the motion output is amplitude-limited, the change in linear velocity between adjacent control cycles does not exceed the maximum allowable acceleration range, and the robot will not stop suddenly or turn sharply.

[0065] As the robot continues to approach the turnover cart, a semi-enclosed area forms between the cart and the right-hand shelf. In the target-approaching reward mode, the robot repeatedly attempts to move towards the final target, but due to unknown occlusion areas and narrow passageways, it repeatedly turns within a short period. The computational processing unit identifies locally optimal trap states within a sliding time window. In this embodiment, the sliding time window is set to... The target distance descent rate threshold is set to The threshold for the number of course reversals is set to Next, the angular velocity symbol switching frequency threshold is set to... The threshold for the continuity of the passable corridor is set to Of the above thresholds, the target distance decrease rate threshold is taken as the [number]th [th ... Percentile value, and adjusted upwards to The threshold for the number of course reversals is determined based on the robot's minimum turning radius; the threshold for the angular velocity symbol switching frequency is less than one-fifth of the control frequency; and the threshold for the continuity of the passable corridor is determined based on the statistical results of the length of the passable grid within the storage channel.

[0066] In this embodiment, when the robot continuously The rate of decrease in the distance to the inner target is The number of course reversals is The frequency of angular velocity symbol switching is [number]. And the continuity of the passable corridor is When the local optimal trap determination condition in formula (3) is met, the computational processing unit determines that the robot has entered the local optimal trap state. This determination is not based solely on the decrease in robot speed, but also on the insufficient decrease in target distance, repeated changes in heading, and instability of the passable corridor. Therefore, it can avoid misjudging the normal deceleration process to avoid pedestrians as a local optimal trap.

[0067] In a locally optimal trap state, the computational processing unit extracts candidate travel directions from the passable corridor layer. In this embodiment, the angular interval of the candidate travel directions is set to... The values ​​are then uniformly converted to radians for calculation. The calculation and processing unit first eliminates collisions with estimated collision times less than [a certain value]. Candidate travel directions are then selected, and those with occlusion uncertainties greater than [value missing] are removed. Candidate travel directions. Occlusion uncertainty threshold. The threshold is determined based on the effective pixel ratio of the depth image, the coverage of the LiDAR, and the intensity of the depth abrupt change at the occlusion boundary behind the turnaround vehicle. Since there are pedestrians and turnaround vehicles obstructing the view in this scene, the threshold is set to a moderately low value to reduce the probability of the robot entering the invisible area.

[0068] For the retained candidate traffic directions, the computational processing unit determines the temporary escape sub-targets according to the lexicographical order rule in formula (4). Specifically, this is done while ensuring that the expected collision time is not less than... The occlusion uncertainty value is no greater than The distance between the candidate escape point and the nearest obstacle boundary is not less than Under the premise of prioritizing the comparison of corridor widths; when the difference in corridor widths between two candidate directions is less than... Then compare the continuous lengths of the passageway; when the difference in continuous lengths is less than... Then, compare the angle between the candidate direction and the final target direction; when the angle difference is less than... Then, the approach speed of the dynamic obstacle is compared. Finally, the computational processing unit selects the location at the robot's left front. Location, closest to the obstacle boundary Continuous length of corridor occlusion uncertainty The grid points are used as temporary escape sub-targets.

[0069] After generating the temporary escape sub-target, the computational processing unit switches the reinforcement learning policy network from target approach reward mode to trap escape reward mode. In trap escape reward mode, the robot no longer forcibly moves towards the final target, but instead prioritizes approaching the temporary escape sub-target and executes actions such as moving away from the turnover vehicle boundary, increasing the width of the passageway, and reducing the number of heading reversals. At this time, the reinforcement learning policy network outputs linear velocity commands. Angular velocity command This allows the robot to navigate away from the semi-enclosed area at a lower speed. This action is converted into the target speed of the left and right drive wheels via differential drive closed-loop control, and corrected through encoder feedback, enabling the robot to smoothly escape the obstacle along the planned trajectory.

[0070] When the robot is less than the temporary escape sub-target The target distance descent rate has recovered to The number of course reversals will be discussed later. Internal reduction to Secondly, the continuity of the passable corridor was restored to [a certain state]. At that time, the computing unit determines that the robot has successfully escaped the obstacle. Escape distance threshold. Based on the radius of the robot's circumcircle and raster resolution Confirmed; the target distance descent rate recovery threshold is taken as the [number]th [value] in the historical normal obstacle avoidance samples. Percentile value; the condition for restoring the continuity of a passable corridor is continuity. The main internal passageway remains connected, and the direction of the main passageway changes by less than [a certain amount]. After successfully escaping the obstacle, the computational processing unit removes the temporary escape sub-target and restores the reinforcement learning policy network to the target-approaching reward mode, enabling the robot to continue running along the warehouse aisle towards the final target storage location.

[0071] During the differential drive closed-loop control phase, the motion controller calculates the target rotational speeds of the left and right drive wheels based on the linear velocity and angular velocity commands output by the reinforcement learning policy network, and reads the actual rotational speeds fed back by the encoders of the left and right drive wheels. When continuously... The actual wheel speed deviation exceeds the target wheel speed within each control cycle. At this time, the motion controller reduces the current linear velocity command and limits the angular velocity command to no more than 1 / 3 of the angular velocity of the previous control cycle. To prevent the robot from slipping, making sharp turns, or colliding near smooth surfaces or obstacle boundaries. In this embodiment, the robot did not experience a sustained deviation in wheel speed during its maneuver around the turnaround vehicle, and ultimately achieved... to The linear speed resumed normal passage and reached the target cargo location.

[0072] As can be seen from this embodiment, the present invention does not rely solely on a reinforcement learning policy network to directly output obstacle avoidance actions. Instead, it first represents static obstacles, dynamic obstacles, unknown occlusions, and passable areas through a locally occupied grid map, and then uses a spatiotemporal collision margin tensor to represent future collision times, occlusion uncertainties, and the continuity of the passable corridor. When the robot encounters insufficient target distance descent, frequent course reversals, or unstable passable corridors, it can identify locally optimal trap states and generate temporary escape sub-targets with safety constraints and passage priorities. Therefore, the robot can achieve continuous, safe, and escape-capable obstacle avoidance path planning in complex environments where warehouse passages, pedestrian traffic, temporary obstacles, and occluded areas coexist.

[0073] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for obstacle avoidance path planning for mobile robots based on reinforcement learning, characterized in that, The method includes: Acquire mobile robot operating environment data, and convert obstacle point cloud data collected by lidar, depth image data collected by depth camera, and robot pose, linear velocity and angular velocity collected by odometry into robot body coordinate system; A local occupancy grid map is constructed based on the obstacle point cloud data and depth image data. The local occupancy grid map consists of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas. Based on the local occupancy grid map and robot pose changes over multiple consecutive frames, the relative motion state of obstacles is estimated, and the robot's trajectory and obstacle occupancy area within the future time window are predicted for the candidate velocity command set, generating a spatiotemporal collision margin tensor. The spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance are input into the reinforcement learning policy network, which outputs linear velocity commands and angular velocity commands. Local optimal trap states are identified based on the target distance descent rate, number of course reversals, frequency of angular velocity sign switching, linear velocity change amplitude, and continuity of passable corridors within a sliding time window. When a local optimal trap state is identified, a temporary escape sub-target is determined according to the spatiotemporal collision margin tensor, and the reinforcement learning policy network is switched from the target approach reward mode to the trap escape reward mode. Linear velocity and angular velocity commands are converted into differential drive control commands, and closed-loop correction is performed in combination with drive wheel encoder feedback. The obstacle avoidance path planning results are also updated cyclically.

2. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 1, characterized in that, The acquisition of mobile robot operating environment data includes: Add acquisition timestamps to obstacle point cloud data, depth image data, and odometry data respectively; Based on the extrinsic parameters of the lidar, the extrinsic parameters of the depth camera, and the installation pose of the odometer, the transformation relationship between the coordinate systems of each sensor and the robot body coordinate system is established; Based on the acquisition time of the odometry data, the obstacle point cloud data and depth image data are time-aligned; When the deviation between any data frame and the reference acquisition time is greater than the synchronization threshold, the data frame is discarded or interpolated for compensation. The time-aligned data is unified into the robot's body coordinate system according to the aforementioned transformation relationship, forming an environmental state input at the same moment.

3. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 2, characterized in that, The construction of the local occupancy raster map includes: Outlier removal and height filtering are performed on obstacle point cloud data to retain point clouds within the robot's collision range; The planned area around the robot is divided into grid cells according to the grid resolution, and the static obstacle area is determined based on the number of point clouds falling into the grid cells and the occupancy probability. Dynamic obstacle candidate regions are determined based on the changes in the occupancy position of the same grid neighborhood in consecutive frames; The unknown occlusion area is determined by the location of the depth abrupt change in the depth image data corresponding to the blind zone, low reflection area or occlusion boundary of the lidar. Grid cells that are not occupied by static obstacle areas, dynamic obstacle candidate areas, or unknown occlusion areas, and whose distance from the obstacle boundary is not less than the robot's outer outline safety distance, are defined as passable areas.

4. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 3, characterized in that, The generation of the spatiotemporal collision margin tensor includes: Based on the position difference of the centroid of the obstacle region in multiple consecutive frames of local occupancy grid map and the robot pose increment, calculate the direction and speed of the obstacle relative to the robot. A set of candidate speed commands is generated based on the robot's maximum linear velocity, maximum angular velocity, maximum acceleration, and minimum turning radius; For each candidate velocity command, calculate the robot's candidate motion trajectory within a future time window according to the discrete predicted step size; Based on the relative direction and speed of the obstacle's movement, generate the predicted area to be occupied by the obstacle within the future time window; Calculate the minimum safe distance between each candidate trajectory and the predicted obstacle occupancy area, the estimated collision time, the dynamic occupancy change, the occupancy uncertainty, and the corridor continuity value; The above calculation results are organized according to the candidate action dimension, time dimension, and space dimension to form a spatiotemporal collision margin tensor consisting of a static occupancy layer, a dynamic residual layer, a collision time layer, an occlusion uncertainty layer, and a passable corridor layer.

5. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 4, characterized in that, The reinforcement learning policy network is trained using a proximal policy optimization algorithm. The training process includes: The state space is defined by the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance. Linear velocity commands and angular velocity commands serve as the continuous motion space; In the target approach reward mode, a reward signal is constructed based on the reduction in target distance, the maintenance of safe distance, the change in path length, the smoothness of motion, and the satisfaction of kinematic constraints. In the trap escape reward mode, a reward signal is constructed based on the approach of the temporary escape sub-target, the distance from the closed obstacle boundary, the increase in the width of the passable corridor, the decrease in the number of course reversals, and the reduction in the risk of approaching dynamic obstacles. The policy network parameters are updated by using a proximal policy optimization algorithm to ensure that the output linear velocity and angular velocity commands satisfy the robot's kinematic constraints.

6. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 5, characterized in that, The identification of locally optimal trap states includes: Calculate the target distance descent rate between the robot and the final target within a sliding time window; The number of times the robot's heading angle change direction reverses within the sliding time window is counted to obtain the heading reversal count; The frequency of angular velocity sign switching is obtained by counting the number of times the sign of the angular velocity changes from positive to negative or from negative to positive within the sliding time window. The difference between the maximum and minimum values ​​of the linear velocity within the sliding time window is calculated to obtain the amplitude of the linear velocity change. The continuity of the accessible corridors is obtained by considering the changes in the connectivity length and direction of the main accessible corridors in the accessible corridor layer. When the target distance descent rate is less than the descent rate threshold, and any one of the following indicators—the number of course reversals, the frequency of angular velocity sign switching, or the continuity of the passable corridor—satisfies the trap threshold condition, the robot is determined to be in a local optimal trap state.

7. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 6, characterized in that, The determination of the temporary escape sub-target includes: Extract candidate travel directions that connect to the robot's current position from the accessible corridor layer; Candidate travel directions with expected collision times less than a safe time threshold are eliminated based on the collision time layer; Candidate travel directions with occlusion uncertainty values ​​greater than the uncertainty threshold are eliminated based on the occlusion uncertainty layer; For the retained candidate travel directions, calculate the corridor width, continuous length, angle with the final target direction, and dynamic obstacle approach speed; Based on the sorting rules of the corridor width, continuous length, angle with the final target direction, and approach speed of dynamic obstacles, the candidate passage direction with the highest priority for getting out of trouble is determined; In the candidate passage direction with the highest escape priority, grid points that are not less than the robot's outer perimeter safety distance from the obstacle boundary and are within the reachable range of the future time window are selected as temporary escape sub-targets.

8. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 7, characterized in that, The reward mode switching includes: When no local optimal trap state is identified, the target-approaching reward mode is maintained, so that the reinforcement learning policy network prioritizes outputting actions that are toward the final target and satisfy the safe distance constraint. After identifying a local optimal trap state and generating a temporary escape sub-target, the trap escape reward mode is activated, causing the reinforcement learning policy network to prioritize outputting actions that approach the temporary escape sub-target, move away from the closed obstacle boundary, and reduce the course reversal. When the distance between the robot and the temporary escape sub-target is less than the escape distance threshold, the target distance decrease rate is greater than the recovery threshold, and the continuity of the passable corridor meets the recovery conditions, the temporary escape sub-target is deleted, and the reinforcement learning policy network is restored to the target approach reward mode.

9. The obstacle avoidance path planning method for mobile robots based on reinforcement learning according to claim 8, characterized in that, The differential drive closed-loop control includes: Calculate the target rotational speeds of the left and right drive wheels based on the linear velocity command, angular velocity command, distance between the left and right drive wheels, and drive wheel radius. Read the actual rotational speeds fed back by the encoders of the left and right drive wheels, and calculate the speed deviations of the left and right drive wheels respectively; The control values ​​of the left and right drive wheel motors are adjusted based on the speed deviations of the left and right drive wheels. The mobile robot is driven to move according to the corrected control values ​​of the left and right drive wheel motors; The robot's pose, velocity, and environmental perception data after movement are fed back into the local occupancy grid map update process, and the spatiotemporal collision margin tensor and obstacle avoidance actions are regenerated until the robot reaches the final goal or receives a stop command.

10. A mobile robot obstacle avoidance path planning system based on reinforcement learning, characterized in that, The system is used to execute the reinforcement learning-based obstacle avoidance path planning method for mobile robots according to any one of claims 1 to 9, the system comprising: The environmental data acquisition module is used to collect obstacle point cloud data, depth image data, robot pose, linear velocity and angular velocity, and unify the collected data into the robot body coordinate system. The local map building module is used to generate a local occupancy raster map consisting of static obstacle areas, dynamic obstacle candidate areas, unknown occlusion areas, and passable areas based on obstacle point cloud data and depth image data. The collision margin generation module is used to generate a spatiotemporal collision margin tensor based on multiple consecutive frames of local occupancy grid map, robot pose changes, and candidate velocity command set. The reinforcement learning decision module is used to input the spatiotemporal collision margin tensor, the robot's current velocity, the target's relative direction, and the target's distance into the reinforcement learning policy network, and output linear velocity commands and angular velocity commands. The trap identification module is used to identify the local optimal trap state based on the target distance descent rate, the number of course reversals, the frequency of angular velocity sign switching, the magnitude of linear velocity change, and the continuity of the passable corridor within the sliding time window. The escape sub-target generation module is used to determine temporary escape sub-targets based on the passable corridor layer, collision time layer, and occlusion uncertainty layer; The reward mode switching module is used to switch the reinforcement learning policy network between the goal approach reward mode and the trap escape reward mode; The differential closed-loop control module is used to convert linear velocity commands and angular velocity commands into control quantities for the left and right drive wheels, and to perform closed-loop correction based on feedback from the drive wheel encoders.

Images