Robotic navigation method and system based on deep reinforcement learning and velocity obstacle

Abstract

The application relates to a robot navigation method and system based on deep reinforcement learning and velocity obstacle, comprising the following steps: acquiring laser radar point cloud data; constructing polar coordinate features according to a current coordinate system and historical frame laser radar point cloud data; obtaining a fusion feature map; obtaining a passable depth probability distribution in each direction according to the fusion feature map, determining reachable points and passable distances, and extracting a passable area; based on a velocity obstacle algorithm, in combination with the passable area, calculating the velocity of the robot relative to the obstacle, defining a collision cone, calculating a velocity obstacle, screening a safe direction, and determining an expected direction; and constructing a reward function according to the expected navigation direction, which is used for guiding the robot to move towards the expected navigation direction. The application combines the advantages of deep reinforcement learning and velocity obstacle, the robot can efficiently move towards the target in a dynamic scene, the shortcomings of falling into a local dilemma and frequent parameter adjustment are avoided, and the overall navigation performance is improved.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to a robot navigation method and system based on deep reinforcement learning and speed obstacles. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Robot navigation technology is an important branch of robotics research, aiming to enable robots to navigate and avoid obstacles efficiently in dynamic and complex environments, completing tasks such as logistics transportation and emergency rescue. Traditional robot navigation methods, such as the Dynamic Window (DWA) method, have been widely used in practical navigation tasks due to their stability and reliability. However, these methods typically rely on static sensor data and require manual parameter adjustments for different environments, resulting in weak adaptability in dynamic environments.

[0004] Deep reinforcement learning, as a self-learning method based on environment interaction, enables end-to-end navigation strategies that decouple perception and control, directly mapping perceptual information into velocity commands. This overcomes the shortcomings of traditional methods, such as being prone to getting stuck in local conditions and requiring frequent parameter adjustments. It allows robots to continuously optimize their navigation strategies, resulting in superior performance in complex scenarios. However, this method suffers from slow convergence speed and high training costs. Therefore, combining deep reinforcement learning with traditional navigation methods holds promise for achieving complementary advantages and improving the robot's adaptability and navigation performance in complex dynamic environments.

[0005] In existing technologies, Patel et al. combined the dynamic constraints of DWA with a path planning method based on deep reinforcement learning to effectively deal with the interference of moving obstacles. Another example is Bektas et al., who proposed to dynamically adjust the control parameters in the artificial potential field algorithm through deep reinforcement learning to adapt to robot navigation tasks in different scenarios. However, such methods usually ignore the consideration of the speed of obstacle movement, and therefore are prone to collisions in complex dynamic environments. Summary of the Invention

[0006] This invention proposes a robot navigation method and system based on deep reinforcement learning and speed obstacle. First, a deep learning algorithm is used to process LiDAR data to extract traversable areas in the environment. Then, the speed obstacle algorithm combines the traversable area to calculate the desired navigation direction and construct a reward function, which is applied to the training process of deep reinforcement learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a robot navigation method based on deep reinforcement learning and speed obstacles, comprising:

[0009] Acquire historical frame LiDAR point cloud data;

[0010] The space around the robot is divided and the current coordinate system is defined. Polar coordinate features are constructed based on the current coordinate system and historical frame LiDAR point cloud data. The polar coordinate features are aligned and weighted and fused to obtain a fused feature map. The passable depth probability distribution in each direction is obtained based on the fused feature map, and the reachable points and passable distances are determined. The passable area is extracted.

[0011] Based on the velocity obstacle algorithm, a collision cone is first defined, then the velocity obstacle space is calculated based on the collision cone, and the desired navigation direction is determined in combination with the passable area. A reward function is constructed based on the desired navigation direction to guide the robot to move in the desired navigation direction.

[0012] A further technical solution is that the specific method for constructing polar coordinate features is as follows: first, the lidar point cloud data is transformed to the current coordinate system, then the lidar point cloud data transformed to the current coordinate system is input into the MLP-Maxpool structure, and finally the polar coordinate features are output.

[0013] A further technical solution utilizes an encoder-decoder structure to perform spatial context encoding on the fused feature map. Specifically, the fused feature map is input into the encoder-decoder structure, which outputs the passable depth probability distribution in each direction. The reachable point in each direction is determined based on the maximum probability in the depth probability distribution, and the passable distance is determined based on the reachable point.

[0014] A further technical solution is that the robot's velocity relative to the obstacle is expressed as: ;in, Indicates the robot's speed. The velocity of the obstacle is represented; the collision cone is defined as:

[0015] ;in, Indicates time interval, This indicates the robot's speed relative to the obstacle. This represents a circular area defined by obstacles.

[0016] A further technical solution is that if the robot's velocity relative to the obstacle is outside the collision cone, then the robot will not collide. Therefore, the velocity obstacle is represented as:

[0017] ;in, Minkowski and, Indicates a speed obstacle.

[0018] A further technical solution involves the following method for selecting safe directions and determining the desired direction: First, directions heading towards the inside of the speed obstacle are excluded, while directions heading towards the outside of the speed obstacle are retained. Specifically, when the robot's speed relative to the obstacle is within the range of [θ-β, θ+β], the corresponding direction is excluded, and the remaining direction is the navigable direction. Then, a semi-circular region centered on the robot's center is defined, and directions in which the robot interacts with the curb and obstacles within this semi-circular region are excluded. Finally, the remaining directions are selected based on sub-target points to obtain the desired direction.

[0019] A further technical solution involves constructing a reward function based on the desired navigation direction, specifically expressed as follows:

[0020] ;

[0021] in, Represents the reward function, This indicates the permissible directional deviation. Indicates the desired navigation direction angle. This represents the robot's linear velocity. This represents the distance between the robot and the forward sub-target point. This represents the maximum value between the robot and the forward sub-target point.

[0022] Secondly, the present invention provides a robot navigation system based on deep reinforcement learning and speed obstacles, comprising:

[0023] The data acquisition module is configured to acquire historical frame LiDAR point cloud data.

[0024] The passable area extraction module is configured to: divide the space around the robot and define the current coordinate system; construct polar coordinate features based on the current coordinate system and historical frame LiDAR point cloud data; align and weight the polar coordinate features to obtain a fused feature map; obtain the passable depth probability distribution in each direction based on the fused feature map, determine the reachable point and passable distance, and extract the passable area;

[0025] The desired navigation direction acquisition and reward function design module is configured to: first define a collision cone based on a velocity obstacle algorithm, then calculate the velocity obstacle space based on the collision cone, and combine it with the passable area to determine the desired navigation direction; and construct a reward function based on the desired navigation direction to guide the robot to move in the desired navigation direction.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] 1. This invention utilizes deep learning algorithms to process LiDAR data to extract passable areas, and then combines them with a speed obstacle algorithm to calculate the desired navigation direction. It fully considers the movement speed and trend of obstacles, and overcomes the problems of traditional methods that rely on static data, have weak adaptability, and ignore the movement speed of obstacles in some combined schemes, enabling the robot to adapt better in dynamic environments.

[0028] 2. This invention obtains the desired direction through the speed barrier algorithm and constructs a reward function based on the desired direction. This function is then applied to the training process of deep reinforcement learning, which improves the slow convergence speed and high training cost of deep reinforcement learning and enhances training efficiency.

[0029] 3. This invention utilizes a velocity obstacle algorithm to construct a safe velocity space. The reward function guides the robot to move in the desired navigation direction. Simultaneously, it comprehensively considers obstacles and sub-targets, encouraging the robot to avoid dynamic obstacles. This ensures continuous and safe navigation of the robot in complex dynamic environments, improving the safety and robustness of the navigation system. Combining the advantages of deep reinforcement learning and velocity obstacles, this invention enables the robot to perform navigation tasks more stably in dynamic scenes, efficiently moving towards the target. It avoids the shortcomings of traditional methods, such as being prone to getting stuck in local dilemmas and requiring frequent parameter adjustments, thus improving overall navigation performance. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0031] Figure 1 This is a flowchart of the robot navigation method based on deep reinforcement learning and speed obstacles of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating the interaction between the passable area and the speed barrier in this invention;

[0033] Figure 3 This is a schematic diagram illustrating the principle of the speed obstacle algorithm of this invention, which combines the speed obstacle algorithm with the passable area to calculate the desired navigation direction. Detailed Implementation

[0034] Example 1

[0035] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] To achieve the above objectives, this embodiment provides a robot navigation method based on deep reinforcement learning and speed obstacles, such as... Figure 1As shown, the specific steps of the method are as follows:

[0037] S1: Acquire historical frame LiDAR point cloud data.

[0038] In step S1, LiDAR point cloud data of the robot's surrounding environment is collected by multi-line LiDAR for subsequent processing such as extraction of passable areas. The acquired LiDAR point cloud data is historical frame LiDAR point cloud data (which is multi-frame LiDAR point cloud data).

[0039] S2: Divide the space around the robot and define the current coordinate system. Construct polar coordinate features based on the current coordinate system and historical frame LiDAR point cloud data. Align and weight the polar coordinate features to obtain a fused feature map. Obtain the passable depth probability distribution in each direction based on the fused feature map, determine the reachable point and passable distance, and extract the passable area.

[0040] In step S2, the first step is to divide the space: the space around the robot is defined as a cylinder centered on the lidar, with radius R and height H. This space is then uniformly divided into... A sector-shaped column. Among them... and They represent along shaft and The number of axes divided, and the corresponding resolution. , This method of spatial division also determines the structural form of spatial features.

[0041] Then comes historical frame point cloud transformation and feature construction: acquiring history Frames of lidar point cloud data, historical The LiDAR point cloud data is transformed to the current coordinate system through a transformation between historical frame poses and the current frame pose. Each frame of LiDAR point cloud data is input into an MLP-Maxpool structure. + 1 frame of LiDAR point cloud data to construct + 1 multidimensional polar coordinate feature.

[0042] Next is feature fusion: Each polar coordinate feature is input into the SAM (SpatialAlignmentModule) module, which aligns and weights the polar coordinate features from different time frames to generate a temporally enhanced fused feature map for subsequent spatial feature extraction.

[0043] Finally, spatial feature extraction and traversable region determination are performed: using an encoder-decoder structure similar to U-Net, spatial context encoding is applied to the fused feature map. Specifically, the fused feature map is input into the encoder-decoder structure, which outputs the traversable depth probability distribution in each direction. The reachable point in each direction is determined based on the maximum probability in the depth probability distribution. Specifically, 384 reachable points are output. The traversable distance is determined based on the reachable points. Specifically, the distance from the robot to these reachable points is the traversable distance, and the traversable region is the area enclosed by these reachable points.

[0044] S3: Based on the speed obstacle algorithm, first define the collision cone, then calculate the speed obstacle space based on the collision cone, and combine it with the passable area to determine the desired navigation direction; construct a reward function based on the desired navigation direction to guide the robot to move in the desired navigation direction.

[0045] In step S3, the working process of the speed obstacle algorithm is represented as follows:

[0046] First, define a circular region. ,by With center at and radius at, The circular area represents the space shared by the obstacle and the robot. Indicates an obstacle. Then Relative to obstacles The speed is: Then, define a collision cone: In the future t Internal meeting and Intersection, or collision. Therefore The outside is safe.

[0047] Then, based on the above, only speed is considered. Will speed toward the obstacle Move in the direction of , and the distance moved is equal to the magnitude of the speed. After moving, obtain , In the future t Internal meeting and Intersection, or collision. Therefore If facing The outer area is safe. When choosing a speed, The velocity direction in the outer region can be selected.

[0048] Based on the working process of the speed obstacle algorithm described above, once a collision-free speed space is calculated, the robot can ensure that there is no collision by selecting any speed within that collision-free space.

[0049] like Figure 3 As shown, according to the obstacles Define a circular region ,by With center at and radius at, . To represent a pedestrian or other vehicle (obstacle), the radius of the circular area changes accordingly. For vehicles, half of their lateral width is taken as the radius.

[0050] The robot's velocity relative to the obstacle is expressed as: ;in, Indicates the robot's speed. This represents the velocity of the obstacle. The collision cone is defined as:

[0051] ;in, Indicates time interval, This indicates the robot's speed relative to the obstacle. This represents a circular area defined by obstacles.

[0052] Therefore, if the robot's velocity relative to the obstacle is within the collision cone... Externally, the robot experiences no collisions, therefore the speed obstacle is represented as:

[0053] ;in, Minkowski and the above process represent the process of... speed toward the obstacle The speed moves in the direction of the velocity, and the distance traveled is equal to the magnitude of that speed. This indicates a speed obstacle, meaning that if the robot maintains its current speed, it will collide with the obstacle in the future.

[0054] After calculating the speed obstacle, safe directions are selected to determine the desired direction. Specifically, directions heading towards the inside of the speed obstacle are first eliminated, while directions heading towards the outside of the speed obstacle are retained. Specifically, when the robot's speed relative to the obstacle is within the range [θ-β, θ+β], the corresponding direction is eliminated, and the remaining directions are navigable directions. Figure 3 It shows and The geometric relationship is calculated using the following formula:

[0055] ; ;

[0056] in, and The geometric relationship between the two included angles is as follows: Figure 3 As shown, It is the spatial angle occupied by the collision cone. In the coordinate system, For obstacles Distance from the origin Towards distance, Obstacles Distance from the origin Distance. This can be understood as The coordinates. Location, express Distance from origin The distance. robot radius, The radius.

[0057] Then define a semi-circular region centered on the robot's center, with the center point being... , indicating a directional range In this embodiment, the radius of the first semicircular area is 5 meters, which serves as a safety threshold to exclude directions in which the robot interacts with the curb and obstacles within the first semicircular area; the remaining directions are selected as the desired directions based on the sub-target points.

[0058] The remaining directions are safe and passable. If the sub-target point's angle relative to the robot... Retained, directly take the desired navigation direction. A sub-target point is selected every 10 meters between the robot and the finish line, and the robot moves towards each sub-target point in turn. Conversely, starting from a sub-target point, the robot traverses to both sides along the passable area point set until a valid angle is obtained. Then set it to This is the final desired navigation direction.

[0059] Based on the desired navigation direction Construct the reward function, specifically as follows:

[0060] ;in, Represents the reward function, This indicates the allowable directional deviation, which is set to 0.5 in this embodiment. Indicates the desired navigation direction angle. This represents the robot's linear velocity. This represents the distance between the robot and the forward sub-target point. This represents the maximum distance between the robot and the forward sub-target point. A sub-target point is taken every 10 meters from the starting point to the final destination. .

[0061] The reward function described above comprehensively considers both obstacles and sub-targets, encouraging the robot to steadily move towards the sub-target point while avoiding dynamic obstacles. The optimal navigation strategy is obtained by continuously training and optimizing the navigation strategy using the reward function.

[0062] Example 2

[0063] This embodiment provides a robot navigation system based on deep reinforcement learning and speed obstacles, which specifically includes the following modules:

[0064] The data acquisition module is configured to acquire historical frame LiDAR point cloud data.

[0065] The passable area extraction module is configured to: divide the space around the robot and define the current coordinate system; construct polar coordinate features based on the current coordinate system and historical frame LiDAR point cloud data; align and weight the polar coordinate features to obtain a fused feature map; obtain the passable depth probability distribution in each direction based on the fused feature map, determine the reachable point and passable distance, and extract the passable area;

[0066] The desired navigation direction acquisition and reward function design module is configured to: first define a collision cone based on a velocity obstacle algorithm, then calculate the velocity obstacle space based on the collision cone, and combine it with the passable area to determine the desired navigation direction; and construct a reward function based on the desired navigation direction to guide the robot to move in the desired navigation direction.

[0067] The implementation of the specific modules in this embodiment is the same as the steps of the robot navigation method based on deep reinforcement learning and speed obstacles described in Embodiment 1, and will not be described in detail here.

[0068] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A robot navigation method based on deep reinforcement learning and speed obstacle, characterized in that, include: Acquire historical frame LiDAR point cloud data; The space around the robot is divided and the current coordinate system is defined. Polar coordinate features are constructed based on the current coordinate system and historical frame LiDAR point cloud data. The specific method for constructing polar coordinate features is as follows: first, the historical frame LiDAR point cloud data is transformed to the current coordinate system, then the LiDAR point cloud data transformed to the current coordinate system is input into the MLP-Maxpool structure, and finally the polar coordinate features are output. The polar coordinate features are aligned and weighted to obtain a fused feature map. The fused feature map is then spatially context-encoded using an encoder-decoder structure. Based on the fused feature map, the passable depth probability distribution in each direction is obtained, and the reachable points and passable distances are determined. Passable regions are then extracted. Specifically, the fused feature map is input into the encoder-decoder structure, which outputs the passable depth probability distribution in each direction. The reachable points in each direction are determined based on the maximum probability in the depth probability distribution, and the passable distances are determined based on the reachable points. Based on the velocity obstacle algorithm, a collision cone is first defined, then the velocity obstacle space is calculated based on the collision cone, and combined with the passable area, the desired navigation direction is determined. A reward function is then constructed based on the desired navigation direction to guide the robot to move in that direction, specifically expressed as follows: ； in, Represents the reward function, Indicates the permissible directional deviation. Indicates the desired navigation direction angle. This represents the robot's linear velocity. This represents the distance between the robot and the forward sub-target point. This represents the maximum distance between the robot and the forward sub-target point.

2. The robot navigation method based on deep reinforcement learning and speed obstacle as described in claim 1, characterized in that, The robot's velocity relative to the obstacle is expressed as: ;in, Indicates the robot's speed. The velocity of the obstacle is represented; the collision cone is defined as: ;in, Indicates time interval, This indicates the robot's speed relative to the obstacle. This represents a circular area defined by obstacles.

3. The robot navigation method based on deep reinforcement learning and speed obstacle as described in claim 2, characterized in that, If the robot's velocity relative to the obstacle is outside the collision cone, then the robot will not collide. Therefore, the velocity obstacle is represented as: ;in, Minkowski and, Indicates a speed obstacle.

4. The robot navigation method based on deep reinforcement learning and speed obstacle as described in claim 1, characterized in that, The specific method for screening safe directions and determining the desired navigation direction is as follows: First, directions towards the inside of the speed obstacle are eliminated, while directions towards the outside of the speed obstacle are retained. Specifically, when the robot's speed relative to the obstacle is within the range of [θ-β, θ+β], the corresponding direction is eliminated, and the remaining directions are navigable directions. Then, a semi-circular region centered on the robot's center is defined, and directions in which the robot interacts with the curb and obstacles within this semi-circular region are eliminated. Finally, the remaining directions are filtered based on sub-target points to obtain the desired direction. Here, θ and β are two included angles, and [θ-β, θ+β] is the spatial angle occupied by the collision cone.

5. A robot navigation system based on deep reinforcement learning and speed obstacle avoidance, characterized in that, include: The data acquisition module is configured to acquire LiDAR point cloud data; The passable area extraction module is configured to: divide the space around the robot and define the current coordinate system, and construct polar coordinate features based on the current coordinate system and historical frame LiDAR point cloud data; the specific method for constructing polar coordinate features is as follows: first, transform the historical frame LiDAR point cloud data to the current coordinate system, then input the transformed LiDAR point cloud data to the MLP-Maxpool structure, and finally output the polar coordinate features; The polar coordinate features are aligned and weighted to obtain a fused feature map. The fused feature map is then spatially context-encoded using an encoder-decoder structure. Based on the fused feature map, the passable depth probability distribution in each direction is obtained, and the reachable points and passable distances are determined. Passable regions are then extracted. Specifically, the fused feature map is input into the encoder-decoder structure, which outputs the passable depth probability distribution in each direction. The reachable points in each direction are determined based on the maximum probability in the depth probability distribution, and the passable distances are determined based on the reachable points. The desired navigation direction acquisition and reward function design module is configured as follows: Based on the velocity obstacle algorithm, first, a collision cone is defined; then, the velocity obstacle space is calculated based on the collision cone; and combined with the passable area, the desired navigation direction is determined. A reward function is constructed based on the desired navigation direction to guide the robot to move in that direction, specifically as follows: ； in, Represents the reward function, Indicates the permissible directional deviation. Indicates the desired navigation direction angle. This represents the robot's linear velocity. This represents the distance between the robot and the forward sub-target point. This represents the maximum distance between the robot and the forward sub-target point.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the robot navigation method based on deep reinforcement learning and velocity obstacles as described in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the robot navigation method based on deep reinforcement learning and velocity obstacles as described in any one of claims 1-4.

Images