A path planning method for unmanned vehicles based on deep reinforcement learning and A-star algorithm

Abstract

This invention discloses a path planning method for unmanned vehicles based on deep reinforcement learning and the A-star algorithm, belonging to the field of unmanned vehicle technology. The method involves driving an unmanned vehicle in an off-road environment to collect 3D point cloud data and record the vehicle's travel path. Multiple grid obstacle maps are generated from the collected point cloud data, and target points in the current grid map are generated from the path traveled by the unmanned vehicle. A simulation environment is constructed using the grid obstacle maps and a rectangle representing the unmanned vehicle. A guide path is generated from each obstacle map using the A-star algorithm. Different difficulty levels are set for different obstacle maps based on the length and smoothness of the guide path. A reward function is designed using the guide path. A proximal policy optimization algorithm and reward function are used to train the unmanned vehicle agent in a simulation environment using a deep neural network, data augmentation, and curriculum learning. The path traveled by the unmanned vehicle is used as the path planning result for the corresponding obstacle map. This method is suitable for path planning for unmanned vehicles in various off-road scenarios and of different sizes.

Description

Technical Field

[0001] The present invention belongs to the field of driverless car technology, and in particular relates to a path planning method for a large-scale off-road environment with limited positioning and no global information. Background Art

[0002] Autonomous driving is a long-term goal in the field of artificial intelligence, requiring a level of intelligence exceeding that achieved by all AI technologies to date. The challenge of autonomous driving lies in ensuring that the unmanned vehicle has an understanding of the environment and its dynamics, enabling it to select the optimal action at every moment through behavioral decision-making and safely and efficiently navigate to its destination. Path planning is a key component of unmanned vehicles, and most methods rely on high-definition maps and high-precision positioning systems. Current path planning algorithms primarily include global path planning and local path planning. Global path planning algorithms include Dijkstra's algorithm (Dijkstra E WA note on two problems in connexion with graphs[J].) and the A-star search algorithm (Hart PE, Nilsson NJ, Raphael BA formally based on the heuristic determination of minimum cost paths[J].), which require accurate environmental models and prior information, such as satellite images and topological road networks. Local path planning algorithms rely on the performance of perception algorithms and sensor accuracy, and their planning results are often not globally optimal. Due to the lack of prior information and complex road conditions, the high cost of maintaining high-definition maps and the lack of accurate positioning systems, these methods cannot be directly applied to complex and unknown environments such as rural and off-road environments.

[0003] On the other hand, deep reinforcement learning-based methods are becoming increasingly popular for solving real-world decision-making tasks due to their ability to achieve end-to-end control without prior knowledge. However, applying deep reinforcement learning to autonomous driving in complex off-road environments still faces numerous challenges. First, autonomous vehicles cannot be directly trained iteratively using deep reinforcement learning methods in real-world environments due to the high mechanical losses. Furthermore, many end-to-end deep reinforcement learning methods use images as input, which lacks generalization capabilities. Furthermore, the visual appearance of off-road environments varies greatly depending on the season, making agents trained in specific scenarios unusable in other settings.

[0004] Therefore, path planning methods in large-scale off-road environments with limited prior information have important research value. To address the shortcomings of existing methods, this paper proposes a path planning method based on deep reinforcement learning and the A-star search algorithm. Summary of the Invention

[0005] The purpose of the present invention is to provide a path planning method for an unmanned vehicle in a large-scale off-road environment with limited prior information. This method does not require a priori high-precision metric maps or topological maps, nor does it require the unmanned vehicle to be directly trained in the off-road environment. It has a certain generalization ability and is applicable to different off-road scenarios and unmanned vehicles of different sizes.

[0006] To achieve the above objectives, the present invention proposes a path planning method based on deep reinforcement learning and A-star search algorithm, comprising the following steps:

[0007] S1: Drive the unmanned vehicle in an off-road environment and collect 3D point cloud data to record the path the unmanned vehicle has traveled;

[0008] S2: Generate multiple grid obstacle maps from the point cloud data collected in S1, and generate target points in the current grid map from the path traveled by the unmanned vehicle;

[0009] S3: Use the grid obstacle map in S2 and a rectangle representing the unmanned vehicle to build a simulation environment. The rectangle is marked with an arrow to indicate the direction of the unmanned vehicle.

[0010] S4: Generate a guided path from each obstacle map in S2 using the A-star algorithm;

[0011] S5: Set different difficulty levels for different obstacle maps based on the length and smoothness of the guidance path in S4;

[0012] S6: Design a reward function using the guided path in S4;

[0013] S7: Using the proximal policy optimization algorithm and the reward function in S6, a deep neural network with data augmentation and curriculum learning is used to train the autonomous vehicle agent in the simulation environment in S3.

[0014] S8: Use the path taken by the unmanned vehicle in S7 as the path planning result corresponding to the obstacle map.

[0015] Preferably, step S2 further comprises:

[0016] S200: Using a ground segmentation algorithm based on Gaussian process regression, the 3D point cloud data within a 100m x 100m area centered on the vehicle is converted into a 500x500 2D grid obstacle map. The resolution of each grid is 0.2m.

[0017] S201: Crop the obstacle map to a size of 256×256 to facilitate deep neural network training.

[0018] S202: The last point of the path traversed in the current obstacle map is taken as the target point.

[0019] Preferably, step S3 further comprises:

[0020] S300: Using Pygame, we used the obstacle map from S2 as the environment and rectangles to represent the autonomous vehicle. We then built a simple simulation environment for manipulating the autonomous vehicle within the obstacle map. Points were drawn at the coordinates of obstacles, while no images were drawn in the traversable areas. Target points were represented by stars. Screenshots of the simulation environment were used as training data for the deep reinforcement learning algorithm.

[0021] S301: Whenever the autonomous vehicle performs an action, the game interface will be refreshed and the autonomous vehicle will be drawn at the new coordinates. The linear acceleration of the autonomous vehicle is l a , the angular velocity is a v , the current angle O, then the coordinates of the unmanned vehicle are calculated as:

[0022] x=x+l a ×cos(O+a v ),

[0023] y=yl a ×sin(O+a v ).

[0024] Preferably, step S4 further comprises:

[0025] S400: Record the coordinates of each obstacle in the obstacle map in S2, set the expansion coefficient according to the environment, and perform obstacle expansion.

[0026] S401: Redraw the obstacle map using the expanded obstacles and save the obstacle coordinates.

[0027] S402: Execute the A-star algorithm in the new obstacle graph to obtain and record each shortest path.

[0028] S403: Evenly select 10 points including the starting point and the target point from the shortest path and use them as the guide path.

[0029] Preferably, step S5 further comprises:

[0030] S500: Calculate the average length of all paths searched by A star and the average value of the curvature

[0031] S501: Using the average path length in S500 and the curvature mean Calculate the difficulty of each shortest path using:

[0032]

[0033] Among them, i is the sequence number of the obstacle graph and the shortest path, L i represents the length of the i-th path, C i represents the curvature of the i-th path.

[0034] Preferably, step S6 further includes:

[0035] S600: Calculate the difference δ between the distance between the current coordinates of the unmanned vehicle and the target point and the distance between the coordinates of the previous moment and the target point goal .

[0036] S601: Calculate the difference δ between the distance between the current coordinates of the unmanned vehicle and the target point and the distance between the coordinates of the previous moment and the target point start .

[0037] S602: Calculate the distance between the current position of the unmanned vehicle and each point on the guidance path, and select the shortest distance as d min .

[0038] S603: Designing reward function R based on guided path astar :

[0039]

[0040] Where W v is the width of the unmanned vehicle.

[0041] S604: The current angle of the unmanned vehicle is O t Angle O with the previous moment t-1 The absolute value of the difference is used to design the reward function R orient :

[0042] R orient =-m×|O t -O t-1 |

[0043] Where m is a constant term used to balance R orient size.

[0044] S605: The final reward function is:

[0045] R=R astar +R orient +c

[0046] Where c is a constant term that rewards or penalizes each action performed by the autonomous vehicle.

[0047] Preferably, step S7 further includes:

[0048] S700: Use a screenshot of the simulation environment as the state space.

[0049] S701: The action space of the unmanned vehicle is a two-dimensional continuous action space with a range of [0,1]. Each dimension represents the linear acceleration l of the unmanned vehicle. a and angular velocity a v .

[0050] S702: Use data augmentation techniques to process neural network input data.

[0051] S703: Set the PPO algorithm hyperparameters based on experience.

[0052] S704: Build a deep neural network with an Actor-Critic structure to train reinforcement learning agents.

[0053] S705: Use the reward function defined in S6 and adopt the course learning method to perform training.

[0054] The effective benefits of this invention compared to the prior art are:

[0055] This paper proposes a path planning method based on deep reinforcement learning and the A-star algorithm, which can be used in complex outdoor environments where prior information is unknown. This method expands the application scenarios of autonomous vehicles, further promotes autonomous driving technology, and generates greater economic benefits.

[0056] 2. The present invention uses a simulation environment to train unmanned vehicles, overcoming the difficulties of training reinforcement learning agents in the real world.

[0057] 3. The path planning results of the present invention are safer, smoother, and more consistent with the dynamic model of a real unmanned vehicle than existing methods, and can be directly applied to unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a framework for an unmanned vehicle path planning method based on deep reinforcement learning and the A-star algorithm.

[0059] Figure 2 This is the corresponding off-road scenario in the specific embodiment of the present invention;

[0060] 2(a) Schematic diagram of the turning scene;

[0061] 2(b) Schematic diagram of obstacle avoidance scenario;

[0062] 2(c) Schematic diagram of a straight road;

[0063] Figure 3 A schematic diagram of a method for generating a grid obstacle map in the present invention;

[0064] Figure 4 This is an obstacle map used to build a simulation environment in a specific embodiment of the present invention;

[0065] Figure 5 The deep neural network architecture used in the present invention;

[0066] Figure 6 is the path planning result in the specific example of the present invention;

[0067] Figure 7 This is the path planning result of the A-star algorithm in the specific example of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described below in detail using specific examples and in conjunction with the accompanying drawings. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through other different examples, and the details in this specification may be modified and altered based on different perspectives and applications without departing from the spirit of the present invention.

[0069] Figure 1 A path planning method for an unmanned vehicle based on deep reinforcement learning and the A-star algorithm is provided in an embodiment of the present invention, comprising the following steps:

[0070] S1: Drive the unmanned vehicle in an off-road environment and collect 3D point cloud data to record the path the unmanned vehicle has traveled.

[0071] Specifically, in this embodiment, the unmanned vehicle travels at a speed of 15 km / h for 300 seconds. Figure 2 Point cloud data was collected in the off-road environment shown. The off-road environment includes driving scenarios such as straight driving, turning, and obstacle avoidance. The driving distance exceeds 1000 meters.

[0072] S2: Generate multiple raster obstacle maps from the point cloud data collected in S1. Generate target points from the path the unmanned vehicle has traveled.

[0073] Specifically, in this embodiment, a ground segmentation method based on Gaussian process regression is used to generate a grid obstacle map. The corresponding relationship between the grid obstacle map and the 3D point cloud is as follows: Figure 3 The step S2 further comprises:

[0074] S200: Using a ground segmentation algorithm based on Gaussian process regression, the 3D point cloud data within a 100m x 100m area centered on the unmanned vehicle is converted into a 500x500 2D grid obstacle map. The coordinates of the unmanned vehicle are at the center of the image. The grid obstacle map has a resolution of 0.2m.

[0075] S201: Generate and save a grid obstacle map every 0.5 seconds from the collected data, and ultimately obtain 600 grid obstacle maps.

[0076] S202: Select 30 representative grid obstacle maps from 600 to build a simulation environment. The selected obstacle maps include straight road, turning, obstacle avoidance and other scenes, such as Figure 4 shown.

[0077] S203: Crop the size of the raster image in S202 to 256×256.

[0078] S3: Build the simulation environment using the grid obstacle map from S2 and a rectangle representing the autonomous vehicle. The rectangle has an arrow on it to indicate the direction of the autonomous vehicle.

[0079] Specifically, the Pygame library is used to treat the grid obstacle map as the simulation environment for the unmanned vehicle. The size of the rectangle is adjusted according to the size of the unmanned vehicle, and the path planning problem is transformed into the problem of operating the unmanned vehicle to reach the target point in the simulation environment. Step S3 further includes:

[0080] S300: Select an obstacle map, convert it into a grayscale map, treat pixels with grayscale values ​​greater than 0 as obstacles and record their coordinates.

[0081] S301: Use the Pygame tool to draw obstacles on the simulation interface.

[0082] S302: The initial coordinates of the unmanned vehicle are the center of the image, the initial angle is random, and the unmanned vehicle is drawn on the simulation interface.

[0083] S303: Use a star to represent the target point and draw it on the simulation interface. The above operations build a simple simulation environment, such as Figure 5 shown.

[0084] S4: Generate a guided path from each obstacle map in S2 using the A-star algorithm.

[0085] Specifically, the A-star algorithm is used to find the shortest path in each obstacle map, and 10 points are extracted from the shortest path as the guide path of the unmanned vehicle. Step S4 further includes:

[0086] S400: Record the coordinates of the obstacles in the obstacle map and expand the obstacles. The expansion coefficient is 4, that is, the original size of the obstacle is 1×1 and the size is 4×4 after expansion.

[0087] S401: Draw an expanded obstacle map in the simulation environment and record the expanded obstacle coordinates using the method in S300.

[0088] S402: Using the A-star algorithm to search for the shortest path from the starting point to the target point in the expanded obstacle graph.

[0089] S402: Evenly sample 10 points including the starting point and the end point from the shortest path as the guide path.

[0090] S5: Set different difficulty levels for different obstacle maps based on the length and smoothness of the shortest path in S4.

[0091] Specifically, this step assigns different levels of difficulty to the corresponding obstacle map based on the length and smoothness of each shortest path, which will be used in subsequent course learning and training methods. Step S5 further includes:

[0092] S500: Calculate the average length of all shortest paths and the average value of the curvature

[0093] S501: Calculate the difficulty of each shortest path using the average path length and average curvature values ​​in S500. The calculation method is:

[0094]

[0095] Among them, i represents the sequence number of the obstacle graph and the shortest path, L i represents the length of the i-th path, C i represents the curvature of the i-th path.

[0096] S6: Design the reward function using the guided path in S4.

[0097] S600: Calculate the difference δ between the distance between the current coordinates of the unmanned vehicle and the target point and the distance between the coordinates of the previous moment and the target point goal Calculate the difference δ between the distance between the current coordinate of the unmanned vehicle and the target point and the distance between the coordinate of the previous moment and the target point start Calculate the shortest distance d between the current position of the unmanned vehicle and the 10 points on the guidance path min The width of the unmanned vehicle is W v .

[0098] S601: The reward function for the autonomous vehicle's driving path is designed based on the guided path:

[0099]

[0100] Where W v is the width of the unmanned vehicle.

[0101] S602: The current angle of the unmanned vehicle is O t Angle O with the previous moment t-1 The absolute value of the difference is used to design the reward function for the angle of the unmanned vehicle:

[0102] R orient =-3×|O t -O t-1 |

[0103] S603: The final reward function is:

[0104] R=R astar +R orient -1.2

[0105] The 1.2 is the penalty for executing each action, the purpose of which is to make the final path as short as possible.

[0106] S7: Using the Proximal Policy Optimization (PPO) algorithm and the reward function in S6, we train the autonomous vehicle agent in the simulation environment in S3 using deep neural networks, data augmentation, and curriculum learning techniques.

[0107] The step S7 further comprises:

[0108] S700: The reinforcement learning observation is a screenshot of the simulation environment in S3, with a size of 3×256×256. The action space of the unmanned vehicle is a two-dimensional continuous action space with a range of [0,1]. Each dimension represents the linear acceleration l of the unmanned vehicle. a and angular velocity a v The reward function is defined in step S603.

[0109] S701: Use data augmentation techniques to process observations. First, randomly crop the 3×256×256 image to 3×128×128. Then add Gaussian noise to the image.

[0110] S702: Build a deep neural network with an Actor-Critic structure to train the autonomous vehicle. The specific structure is as follows Figure 6 As shown in Figure 1, both the Actor and the Critic are multi-layer perceptrons. The input of the neural network is the observation in S701, and the output is the value and action, which are used to calculate the loss function of the subsequent PPO algorithm.

[0111] S703: Based on experience, the hyperparameters of PPO are set as follows: Adam optimizer, learning rate 1e -5 ; The discount factor γ is 0.9; The entropy weight is 1e -2 ; The weight of the value function is 1e -3 ; The ratio between the old strategy and the new strategy is clipped in the range [0.8, 1.2]

[0112] S704: Training is performed using a curriculum learning approach. During training, as long as the autonomous vehicle obtains the required rewards in a low-difficulty environment, the environment will switch to a higher difficulty level. The difficulty of the environment is calculated in S5.

[0113] S8: Use the path taken by the unmanned vehicle in S7 as the path planning result corresponding to the obstacle map.

[0114] Specifically, whenever the unmanned vehicle reaches the target point, the coordinates of the unmanned vehicle in the obstacle map are recorded and used as the path planning result.

[0115] In order to better illustrate the technical effect of the present invention, the inventor conducted experiments in obstacle avoidance and turning environments. The final results are as follows Figure 7 As shown in the figure, the solid line is the planning result of the present invention, and the dotted line is the planned path of star A. Intuitively, the result of the present invention is farther away from obstacles, so it is safer. The curvature of the planned path is used as the standard for evaluating the smoothness of the path. The experimental results are shown in Table 1:

[0116] Table 1 Example test results

[0117] <![CDATA[Curvature (m -1 )]]> Path planning method of the present invention A star search algorithm Obstacle avoidance environment 0.0136 0.143 Turning environment 0.018 0.562

Claims

1. A path planning method based on deep reinforcement learning and A-star search algorithm, characterized in that: The steps include: S1: Drive the unmanned vehicle in an off-road environment and collect 3D point cloud data to record the path the unmanned vehicle has traveled; S2: Generate multiple grid obstacle maps from the point cloud data collected in S1, and generate target points in the current grid map from the path traveled by the unmanned vehicle; S3: Use the grid obstacle map in S2 and a rectangle representing the unmanned vehicle to build a simulation environment. The rectangle is marked with an arrow to indicate the direction of the unmanned vehicle. S4: Generate a guided path from each obstacle map in S2 using the A-star algorithm; S400: Record the coordinates of each obstacle in the obstacle map in S2, Set the expansion coefficient according to the environment and perform obstacle expansion; S401: Redraw the obstacle map using the expanded obstacles and save the obstacle coordinates; S402: Execute the A-star algorithm in the new obstacle graph to obtain and record each shortest path; S403: evenly selecting 10 points including the starting point and the target point from the shortest path to use as the guide path; S5: Set different difficulty levels for different obstacle maps based on the length and smoothness of the guidance path in S4; S500: Calculate the average length of all paths searched by A star and the average value of the curvature S501: Using the average path length in S500 and the curvature mean Calculate the difficulty of each shortest path using: Among them, i is the sequence number of the obstacle graph and the shortest path, L i represents the length of the i-th path, C i represents the curvature of the i-th path; S6: Design a reward function using the guided path in S4; S600: Calculate the difference δ between the distance between the current coordinates of the unmanned vehicle and the target point and the distance between the coordinates of the previous moment and the target point goal ; S601: Calculate the difference δ between the distance between the current coordinates of the unmanned vehicle and the target point and the distance between the coordinates of the previous moment and the target point start ; S602: Calculate the distance between the current position of the unmanned vehicle and each point on the guidance path, and select the shortest distance as d min ; S603: Designing reward function R based on guided path astar : Where W v is the width of the unmanned vehicle; S604: The current angle of the unmanned vehicle is O t Angle O with the previous moment t-1 The absolute value of the difference is used to design the reward function R orient : R orient =-m×|O t -O t-1 | Where m is a constant term used to balance R orient size; S605: The final reward function is: R=R astar +R orient +c Where c is a constant term that rewards or penalizes each action performed by the autonomous vehicle; S7: Using the proximal policy optimization algorithm and the reward function in S6, a deep neural network with data augmentation and curriculum learning is used to train the autonomous vehicle agent in the simulation environment in S3. S700: Use the screenshot of the simulation environment as the state space; S701: The action space of the unmanned vehicle is a two-dimensional continuous action space with a range of [0,1]. Each dimension represents the linear acceleration l of the unmanned vehicle. a and angular velocity a v ; S702: Use data augmentation techniques to process neural network input data; S703: Setting PPO algorithm hyperparameters based on experience; S704: Build a deep neural network with an Actor-Critic structure to train a reinforcement learning agent; S705: Use the reward function defined in S6 and adopt the course learning method to train; S8: Use the path taken by the unmanned vehicle in S7 as the path planning result corresponding to the obstacle map.

2. A path planning method based on deep reinforcement learning and A-star search algorithm according to claim 1, characterized in that: The step S2 comprises the following steps: S200: Using a ground segmentation algorithm based on Gaussian process regression, the 3D point cloud data within a 100m x 100m area centered on the vehicle is converted into a 500x500 2D grid obstacle map, with a resolution of 0.2m per grid. S201: Crop the obstacle map to a size of 256×256 to facilitate deep neural network training; S202: The last point of the path traversed in the current obstacle map is taken as the target point.

3. A path planning method based on deep reinforcement learning and A-star search algorithm according to claim 2, characterized in that: The step S3 comprises the following steps: S300: Using Pygame, we used the obstacle map from S2 as the environment and rectangles to represent the autonomous vehicle. We then built a simple simulation environment for manipulating the autonomous vehicle within the obstacle map. Points were drawn at the coordinates of obstacles, while no images were drawn in the traversable areas. Stars were used to represent target points. Screenshots of the simulation environment were used as training data for the deep reinforcement learning algorithm. S301: Whenever the unmanned vehicle performs an action, the game interface will be refreshed and the unmanned vehicle will be drawn at the new coordinates. The linear acceleration of the unmanned vehicle is l a , the angular velocity is a v , the current angle O, then the coordinate calculation method of the unmanned vehicle is: x=x+l a ×cos(O+a v ) y=yl a ×sin(O+a v )。

Images