A Path Planning Method for Ground Robots Based on Deep Reinforcement Learning

Abstract

This invention belongs to the field of robot path planning and discloses a ground robot path planning method based on deep reinforcement learning. Based on deep reinforcement learning and combined with a robot dynamics model, it aims to propose a safe and reliable path in a field environment. The method first constructs a 3D scene and preprocesses it, then performs dynamic modeling of the robot to build a deep reinforcement learning model. This model uses the Asynchronous Advantage Actor-Critic method to learn the optimal path planning strategy and the optimal state value function, designs a reward function considering terrain and distance, and designs an exploration strategy based on the interaction between the robot and different terrains. Finally, a deep convolutional neural network model is trained to drive the robot to obtain the maximum reward value during operation. This method comprehensively considers terrain and robot dynamics information to plan a safe and reliable path for the robot in a 3D scene.

Description

Technical Field

[0001] This invention belongs to the field of robot path planning, and specifically describes a ground robot path planning algorithm based on deep reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence, autonomous navigation of ground robots in off-road environments is crucial in many applications, such as military operations, agriculture, and planetary exploration. Compared to paved roads, off-road terrain presents additional challenges to autonomous navigation. In off-road terrain, robots may experience unexpected situations such as rollovers and slippage. These situations arise from a lack of consideration for the interaction between the robot and its environment, leading to failures in path planning and making it difficult to meet the requirements for successful navigation of ground robots in off-road environments. Summary of the Invention

[0003] The purpose of this invention is to provide a safe and reliable deep reinforcement learning path planning method for robots in off-road environments, in order to overcome the shortcomings of the aforementioned robot path planning.

[0004] The technical solution adopted by this invention to achieve the above objectives is: a path planning method for ground robots based on deep reinforcement learning, comprising the following steps:

[0005] The map information in the 3D scene is preprocessed and used as input into the deep reinforcement learning model;

[0006] A deep reinforcement learning model is constructed, and a reward function is designed based on the map's elevation information, terrain type, and robot accessibility constraints to calculate the robot's reward after each action for model training.

[0007] The deep reinforcement learning model is trained to drive the robot to obtain the maximum reward value during operation and obtain the optimal path planning strategy to realize the robot's path planning.

[0008] The preprocessing of map information in the 3D scene includes the following steps:

[0009] The point cloud data acquired by the lidar is converted into elevation data to form an elevation map; the map stores elevation information in each grid cell.

[0010] The geometric traversability cost of constructing a grid is used to calculate the reward function; the geometric traversability cost of each grid is expressed as:

[0011]

[0012] In the formula, T geoThe geometric traversability cost is for each grid cell, where w1, w2, and w3 are weight scaling factors, and T is the weighting factor. h ,T s ,T r These represent the step height cost, slope cost, and terrain smoothness cost for the robot to traverse the current terrain type, respectively. hmax ,T smax ,T rmax These are the robot's maximum step height, slope, and terrain flatness.

[0013] The deep reinforcement learning model is the A3C algorithm model.

[0014] The robot's dynamics information includes the vertical loads on the front and rear wheels of the ground robot; wherein, the robot's rotational speed about an axis perpendicular to the vehicle's centerline. as follows:

[0015]

[0016] L f ,L r These represent the distance from the vehicle's center of gravity to the front axle and the distance from the vehicle's center of gravity to the axle, respectively. δ represents the steering angle, and F... yf F represents the lateral force on the front wheel. yr I represents the lateral force on the rear wheel. zz is the robot's yaw moment of inertia; m is the robot's mass; g is the acceleration due to gravity.

[0017] F yf =ηC αf F zf α f F yr =ηC αr F zr α r

[0018]

[0019]

[0020] In the formula, C αf C αr α represents the cornering stiffness of the front wheel and the cornering stiffness of the rear wheel, respectively. f ,α r F represents the sideslip angle of the front wheel and the sideslip angle of the rear wheel, respectively. zf ,F zr h represents the vertical load on the front wheel and the vertical load on the rear wheel, respectively. c This represents the height of the vehicle's center of gravity above the ground. It is the longitudinal acceleration in the robot coordinate system, η represents the ground friction coefficient, roll represents the roll angle, and pitch represents the pitch angle.

[0021] The reward function is as follows:

[0022] R = R cost +R goal +R move +R fail

[0023] In the formula, R cost It is the cost of the robot moving around in the map, R goal It's the reward for the robot reaching the finish line, R move R is the penalty value after each move of the robot. fail This is the penalty for the robot after it collides with an obstacle or fails to navigate, where R... cost The specific functions are as follows:

[0024]

[0025] In the formula T geo It is the geometric traversability cost of each grid, T robot It is the threshold that allows the robot to pass through at a certain cost;

[0026] Where R move The specific function is as follows:

[0027]

[0028] In the formula R time It is a time penalty factor, representing the time penalty after each move of the robot. This indicates the current position P of the robot. now To the finish line position P goal Euclidean distance.

[0029] In the preprocessing of map information in the 3D scene, the upper bound h is used. ub and lower bound h lb Uniform distribution U(h) lb ,h ub To scale the grid height, the height of each grid cell can be expressed as:

[0030] h = h o U h (h lb ,h ub )

[0031] In the formula, h o It is the original height obtained from the grid;

[0032] The robot's motion during training can be represented as follows:

[0033]

[0034]

[0035] δ=δ+IU δ (δ lb ,δ ub )

[0036] In the formula, I represents the Bernoulli distribution, indicating that motion disturbances can occur randomly during the interaction between the robot and the environment;

[0037] Motion disturbance applied to the longitudinal velocity of the robot lateral speed of the robot And the yaw rotation angle δ, which is determined by the uniform distribution of longitudinal velocity U vx (v xlb ,v xub ), uniform distribution in the lateral direction U vy (v ylb ,v yub Uniform distribution of yaw angle U δ (δ lb ,δ ub ) indicates that v xlb ,v ylb ,δ lb It represents the lower bound, v xub ,v yub ,δ ub It represents the upper realm.

[0038] Training a deep reinforcement learning model to drive the robot to obtain the maximum reward value during operation and to obtain the optimal path planning strategy, thereby realizing the robot's path planning, includes the following steps:

[0039] The elevation map information and robot dynamics information are used as two input branches, which are then merged through a convolutional layer. The merged tensor is then fed into a gated linear unit to capture the robot's previous state information.

[0040] Following the gated linear unit, the two output branches actor and critic respectively output the optimal path planning strategy π(a). t ,s t ;θ), state value function V(s) t ;θ v ); π represents the robot's strategy, and s represents the robot's state space, including the robot's current position information p and the distance d from the robot to the target point. goal; 'a' represents the robot's motion space, including its linear and angular velocities; 't' represents time; 'θ' represents the Actor network weight parameters. v This represents the weight parameters of the Critic network.

[0041] The elevation map information and robot dynamics information are used as two input branches, including the following steps:

[0042] The elevation map is input in the form of a single-layer matrix. This branch value contains 4 convolutional layers, all of which use the ReLU function as the activation function. A max pooling layer is added between two convolutional layers to reduce the dimensionality of the input image.

[0043] Robot dynamics information is taken as input and processed through a fully connected layer whose output matches the output shape of the elevation map branch.

[0044] Multiple pre-built deep reinforcement learning models are trained simultaneously by inputting different starting and target positions of the robots, including the following steps:

[0045] Each model is trained using a separate thread;

[0046] Meanwhile, during the training process of the deep reinforcement learning model, for each deep reinforcement learning model, the state-action step is evaluated, and the weight parameters of the network are continuously updated based on the evaluation value. Ultimately, the update direction of the deep reinforcement learning model tends to maximize the reward, until the reward value of a certain deep reinforcement learning model is within a set range. At this point, the weight parameters of the deep reinforcement learning model are the optimal weight parameters.

[0047] A global network composed of multiple deep reinforcement learning models stores the optimal weight parameters and transmits them synchronously to the deep reinforcement learning models.

[0048] After training the deep reinforcement learning model, robot dynamics information and real-time elevation map are input into the deep reinforcement learning model that stores the optimal weight parameters to obtain the real-time optimal path planning strategy, thereby realizing real-time path planning for the ground robot.

[0049] This invention offers the following advantages and benefits: Compared to existing robot path planning technologies, the algorithm proposed in this invention can handle rugged and complex off-road environments. The algorithm considers the height, slope, and terrain smoothness information of different terrain types on the map and constructs a reward function accordingly. Furthermore, it considers the interaction between the robot and its environment. By introducing the Actor-Critic framework for policy learning and training, online execution avoids the problem of high computational load and requires less computing power. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the path planning method for ground robots based on deep reinforcement learning proposed in Embodiment 1 of the present invention.

[0051] Figure 2 This diagram illustrates the overall framework for path planning of ground robots based on deep reinforcement learning, as proposed in Embodiment 1 of this invention.

[0052] Figure 3 This diagram illustrates A3C, the path planning method for ground robots based on deep reinforcement learning proposed in Embodiment 1 of the present invention.

[0053] Figure 4 This diagram illustrates the network structure of the ground robot path planning method based on deep reinforcement learning proposed in Embodiment 1 of this invention. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 The diagram shows a flowchart of a path planning method for a ground robot based on deep reinforcement learning. This embodiment uses a four-wheel drive ground robot, which is a prior art technology. A path planning method for a ground robot based on deep reinforcement learning includes the following steps:

[0056] S1. Construct a 3D scene and preprocess the map information in the 3D scene.

[0057] S2. Construct a deep reinforcement learning model that uses the Asynchronous Advantage Actor-Critic method to learn the optimal path planning strategy and the optimal state value function.

[0058] S3. Taking into account the map's elevation information, terrain type, and the robot's maximum accessibility, a reward function is designed to calculate the robot's reward after each action. The terrain type is obtained through an RGB camera, such as grass or dirt roads on the map.

[0059] S4. Design exploration strategies to select appropriate actions to deal with the current state.

[0060] S5. Train the deep convolutional neural network model to drive the robot to obtain the maximum reward value during operation.

[0061] In one embodiment, the preprocessed map information in step S1 is as follows:

[0062] like Figure 2As shown, the point cloud data acquired by the LiDAR is converted into elevation data, and the created elevation map stores height information in each grid cell. The geometric traversability cost of each grid cell can be expressed as:

[0063]

[0064] In the formula T geo The geometric traversability cost is for each grid cell, where w1, w2, and w3 are weight scaling factors, and T is the weighting factor. h ,T s ,T r It is the cost of step height, slope, and terrain smoothness, T hmax ,T smax ,T rmax These are the robot's maximum traversal step height, slope, and terrain flatness. Additionally, a maximum traversable cost threshold is set for the robot, and the total cost must be below this threshold.

[0065] In addition, the terrain type is extracted from the RGB camera images from the semantic segmentation module. Each terrain type is predefined in the program, and each pixel value is classified into different types of terrain from predefined terrain categories. Therefore, each grid is labeled with a terrain type.

[0066] In one embodiment, the optimal state value function in step S2 is:

[0067] v * =max π v π (s)

[0068] In the formula v * s represents the optimal state value of the deep reinforcement learning model, that is, the state value that the robot can achieve with the greatest value among all policies π, where π represents the robot's policy and s represents the robot's state space.

[0069] The state space is as follows:

[0070] s=[p,d goal ]

[0071] In the formula, p represents the robot's current position information, and d goal This represents the distance from the robot to the target point.

[0072] The optimal strategy is:

[0073]

[0074] In the formula, π *(a|s) represents the optimal strategy of the deep convolutional neural network model, s represents the robot's state space, a represents the robot's action space, including the robot's linear and angular velocities, A represents the set of all robot actions, and q * (s,a) represents the optimal action value, which can be expressed by the following formula:

[0075] q * (s,a)=max(q π (s,a))

[0076] In the formula, q * (s,a) represents the optimal action value, which is the reward value after the robot makes a decision by taking the optimal strategy in state S and executing action a.

[0077] In one embodiment, the Asynchronous Advantage Actor-Critic (A3C) in step S2 is explained as follows:

[0078] like Figure 3 As shown, the Asynchronous Advantage Actor-Critic method used in S2 is a deep reinforcement learning framework based on asynchronous gradients. It leverages the parallel execution of multiple threads, allowing multiple actors to train in parallel and periodically synchronize with global parameters. In initializing the discrete time step t, in state s... t The robot below follows the strategy π(a) t |s t ;θ) Perform actions to transform the robot into s t To maximize expected future returns. Strategy π(a) t |s t The state value function V(s) is determined by a weighted neural network parameterized by θ. Navigation actions include the robot's linear velocity v and angular velocity ω. t ;θ v ) is from state s t The initial expected future cumulative return is calculated using weights θ. v A parameterized state-value network is used for estimation. The reward is used to calculate the reward at each step relative to θ and θ'. v The gradient of the target gradient function is as follows:

[0079]

[0080] In the formula, H is the entropy value that encourages the robot to explore different navigation actions, and β is a hyperparameter that determines the strength of H. Gated recurrent units are used as recurrent layers in the network structure to obtain the robot's previous state information. It is the gradient operator, A(s)t ,a t ;θ v The dominance function is represented by the following formula:

[0081]

[0082] In the formula, Γ is the discount factor, t is the discrete time step, n is the number of steps before the end of training, and r is the reward function.

[0083] In one embodiment, the A3C network architecture in S2 is as follows:

[0084] like Figure 4 As shown, the entire neural network uses elevation maps and robot dynamics information as inputs to determine the final navigation action. There are two input branches, each corresponding to a different input type. The elevation map and robot dynamics information are merged together through a shared convolutional layer. After the branches are merged, the merged tensor is fed into a gated linear unit (GLU) to capture the robot's previous state information. After the GLU, the network splits into two branches, actor and critic. The actor outputs the optimal path planning policy π(a...). t ,s t The critic output is the state value function V(s); θ), t ;θ v ).

[0085] Specifically, the elevation map is input as a single-layer 200×200 matrix. This branch contains four convolutional layers, all using the ReLU activation function. A max pooling layer is added between each convolutional layer to reduce the dimensionality of the input image, with a pooling window size of 2*2. Robot dynamics information is used as input and processed through a fully connected layer; the output of this layer matches the shape of the elevation map branch's output. The data from the two layers are merged and used as input to the next convolutional layer. The output of this convolutional layer is then fed into the GLU to capture the robot's previous state information. After the GLU, the network splits into two branches: actor and critic. The actor outputs the optimal path planning strategy π(a). t ,s t The critic output is the state value function V(s); θ), t ;θ v ).

[0086] The robot's dynamics model in the simulation is based on the Project Chrono high-mobility multi-purpose wheeled vehicle model, where the robot's dynamics model is as follows:

[0087]

[0088] In the formula, It is the longitudinal velocity in the world coordinate system. It is the lateral velocity in the world coordinate system. It is the velocity around the z-axis in the world coordinate system. It is the longitudinal acceleration in the robot's coordinate system. It is the lateral acceleration in the robot's coordinate system. This is the speed at which the robot rotates around an axis perpendicular to the robot's centerline. It describes the robot's lateral motion and stability characteristics during operation. The specific formula is as follows:

[0089]

[0090] In the formula, roll represents the roll angle, and pitch represents the pitch angle. This represents the angular velocity in the robot's coordinate system.

[0091] The specific formula is as follows:

[0092]

[0093] The specific formula is as follows:

[0094]

[0095] Where L f ,L r This represents the distance from the vehicle's center of gravity to the front axle and the axle itself; δ represents the steering angle. This represents the longitudinal velocity in the robot's coordinate system. F represents the velocity around the x-axis in the robot's coordinate system. G F represents the lateral component of the vehicle's weight. yf This represents the lateral force on the front wheel, F. yr This represents the lateral force on the rear wheel, I zz This is the robot's yaw moment of inertia. The specific formula is:

[0096] F G = -mgcos(pitch)cos(roll)

[0097] In the formula, m is the mass of the robot, and g is the acceleration due to gravity.

[0098] F yf =ηC αf F zf α f F yr =ηCαr F zr α r

[0099] In the formula, C αf C αr This represents the cornering stiffness of the front and rear wheels, α. f ,α r This represents the sideslip angle of the front and rear wheels. F zf ,F zr This represents the vertical load on the front and rear wheels, h. c This represents the height of the vehicle's center of gravity above the ground, and η represents the coefficient of friction against the ground. The specific formula is:

[0100]

[0101]

[0102] In one embodiment, the specific reward function in S3 is:

[0103] R = R cost +R goal +R move +R fail

[0104] In the formula R cost It is the cost of the robot moving around in the map, R goal It's the reward for the robot reaching the finish line, R move R is the penalty value after each move the robot makes. fail This is the penalty for the robot after it collides with an obstacle or fails to navigate, where R... cost The specific functions are as follows:

[0105]

[0106] In the formula T geo It is the geometric traversability cost of each grid, T robot It is the threshold that allows the robot to pass through at a certain cost.

[0107] Where R move The specific function is as follows:

[0108]

[0109] In the formula R time It is a time penalty factor, representing the time penalty after each move of the robot. This represents the Euclidean distance from the robot's current position to the destination position.

[0110] In one embodiment, the exploration strategy for deep reinforcement learning designed in step S4 is as follows:

[0111] To address the impacts of robot-ground interactions, two strategies were employed during training to mitigate their effects: the first was variable terrain height, and the second was robot motion disturbances.

[0112] When using LiDAR and RGB cameras to build maps, measurement errors may cause discrepancies between the grid height values ​​and the actual height values, potentially leading to incorrect navigation actions during robot navigation. Therefore, an upper bound h is used. ub and lower bound h lb Uniform distribution U(h) lb ,h ub The height of the grid is scaled using this method. The height of each grid cell can then be expressed as:

[0113] h = h o U h (h lb ,h ub )

[0114] In the formula, h o It is the original height obtained from the grid.

[0115] When a robot navigates through rough terrain, unpredictable movements can occur during its interaction with the ground, such as slipping on mud. Applying motion disturbances to the robot's longitudinal velocity... lateral speed of the robot and yaw rotation angle δ. These motions are caused by a uniformly distributed U vx (v xlb ,v xub ), U vy (v ylb ,v yub ) and U δ (δ lb ,δ ub ) indicates that v xlb ,v ylb ,δ lb It represents the lower bound, v xub ,v yub ,δ ub This represents the upper bound. Therefore, the robot's motion during training can be represented as:

[0116]

[0117]

[0118] δ=δ+IU δ (δ lb ,δ ub )

[0119] In the formula, I represents the Bernoulli distribution, indicating that motion disturbances can occur randomly during the interaction between the robot and the environment.

[0120] In one embodiment, step S5 involves training the robot by generating different starting and target positions in a simulation environment. During neural network training, the state-action pair at each step is evaluated for quality, and the network's weight parameters are continuously updated based on the evaluation values. Ultimately, the neural network updates in the direction of maximizing reward, and after a preset number of training rounds, the reward obtained by the mobile robot in each round becomes relatively stable. To improve training efficiency, nine agent threads are used to run nine simulation experiments simultaneously. The global network stores the optimal weight parameters and transmits them to each local network through a synchronization mechanism. The Actor network is responsible for selecting actions, and the Critic network evaluates the quality of the current state-action pair and updates the parameters of the Actor network based on the evaluation results. After training, the trained network is applied, and the robot can perform path planning actions in the simulation environment.

[0121] The foregoing has provided a detailed description of the off-road vehicle path planning method based on deep reinforcement learning provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention; the descriptions of the above embodiments are merely for the purpose of helping to understand the core ideas of this invention.

Claims

1. A path planning method for ground robots based on deep reinforcement learning, characterized in that, Includes the following steps: The map information in the 3D scene is preprocessed for input into the deep reinforcement learning model; A deep reinforcement learning model is constructed, and a reward function is designed based on the map's elevation information, terrain type, and robot accessibility constraints to calculate the robot's reward after each action for model training. The deep reinforcement learning model is trained to drive the robot to obtain the maximum reward value during operation and obtain the optimal path planning strategy to realize the robot's path planning. The reward function is as follows: R=R cost +R goal +R move +R fail In the formula, R cost It is the cost of the robot moving around in the map, R goal It's the reward for the robot reaching the finish line, R move R is the penalty value after each move of the robot. fail This is the penalty for the robot after it collides with an obstacle or fails to navigate, where R... cost The specific functions are as follows: In the formula T geo It is the geometric traversability cost of each grid, T robot It is the threshold that allows the robot to pass through at a certain cost; Where R move The specific function is as follows: In the formula R time It is a time penalty factor, representing the time penalty after each move of the robot. This indicates the current position P of the robot. now To the finish line position P goal Euclidean distance.

2. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, The preprocessing of map information in the 3D scene includes the following steps: The point cloud data acquired by the lidar is converted into elevation data to form an elevation map; the map stores elevation information in each grid cell. The geometric traversability cost of constructing a grid is used to calculate the reward function; the geometric traversability cost of each grid is expressed as: In the formula, T geo The geometric traversability cost is for each grid cell, where w1, w2, and w3 are weight scaling factors, and T is the weighting factor. h ,T s ,T r These represent the step height cost, slope cost, and terrain smoothness cost for the robot to traverse the current terrain type, respectively. hmax ,T smax ,T rmax These are the robot's maximum step height, slope, and terrain flatness.

3. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning model is the A3C algorithm model.

4. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, The robot's dynamics information includes the vertical loads on the front and rear wheels of the ground robot; wherein, the robot's rotational speed about an axis perpendicular to the vehicle's centerline. as follows: L f ,L r These represent the distance from the vehicle's center of gravity to the front axle and the distance from the vehicle's center of gravity to the axle, respectively. δ represents the steering angle, and F... yf F represents the lateral force on the front wheel. yr I represents the lateral force on the rear wheel. zz is the robot's yaw moment of inertia; m is the robot's mass; g is the acceleration due to gravity. F yf =ηC αf F zf a f ,F yr =ηC αr F zr a r In the formula, C αf C αr α represents the cornering stiffness of the front wheel and the cornering stiffness of the rear wheel, respectively. f ,α r F represents the sideslip angle of the front wheel and the sideslip angle of the rear wheel, respectively. zf ,F zr h represents the vertical load on the front wheel and the vertical load on the rear wheel, respectively. c This represents the height of the vehicle's center of gravity above the ground. It is the longitudinal acceleration in the robot coordinate system, η represents the ground friction coefficient, roll represents the roll angle, and pitch represents the pitch angle.

5. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, In the preprocessing of map information in the 3D scene, the upper bound h is used. ub and lower bound h lb Uniform distribution U(h) lb ,h ub To scale the grid height, the height of each grid cell can be expressed as: h=h o U h (h lb ,h ub ) In the formula, h o It is the original height obtained from the grid; The robot's motion during training can be represented as follows: δ=δ+IU δ (d lb ,d ub ) In the formula, I represents the Bernoulli distribution, indicating that motion disturbances can occur randomly during the interaction between the robot and the environment; Motion disturbance applied to the longitudinal velocity of the robot lateral speed of the robot And the yaw rotation angle δ, which is determined by the uniform distribution of longitudinal velocity U vx (v xlb ,v xub ), uniform distribution in the lateral direction U vy (v ylb ,v yub Uniform distribution of yaw angle U δ (δ lb ,δ ub ) indicates that v xlb ,v ylb ,δ lb It represents the lower bound, v xub ,v yub ,δ ub It represents the upper realm.

6. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, Training a deep reinforcement learning model to drive the robot to obtain the maximum reward value during operation and to obtain the optimal path planning strategy, thereby realizing the robot's path planning, includes the following steps: The elevation map information and robot dynamics information are used as two input branches, which are then merged through a convolutional layer. The merged tensor is then fed into a gated linear unit to capture the robot's previous state information. Following the gated linear unit, the two output branches actor and critic respectively output the optimal path planning strategy π(a). t ,s t ;θ), state value function V(s) t ;θ v ); π represents the robot's strategy, and s represents the robot's state space, including the robot's current position information p and the distance d from the robot to the target point. goal ; 'a' represents the robot's motion space, including its linear and angular velocities; 't' represents time; 'θ' represents the Actor network weight parameters. v This represents the weight parameters of the Critic network.

7. The method for path planning of a ground robot based on deep reinforcement learning according to claim 6, characterized in that, The elevation map information and robot dynamics information are used as two input branches, including the following steps: The elevation map is input in the form of a single-layer matrix. This branch value contains 4 convolutional layers, all of which use the ReLU function as the activation function. A max pooling layer is added between two convolutional layers to reduce the dimensionality of the input image. Robot dynamics information is taken as input and processed through a fully connected layer whose output matches the output shape of the elevation map branch.

8. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, Multiple pre-built deep reinforcement learning models are trained simultaneously by inputting different starting and target positions of the robots, including the following steps: Each model is trained using a separate thread; Meanwhile, during the training process of the deep reinforcement learning model, for each deep reinforcement learning model, the state-action step is evaluated, and the weight parameters of the network are continuously updated based on the evaluation value. Ultimately, the update direction of the deep reinforcement learning model tends to maximize the reward, until the reward value of a certain deep reinforcement learning model is within a set range. At this point, the weight parameters of the deep reinforcement learning model are the optimal weight parameters. A global network composed of multiple deep reinforcement learning models stores the optimal weight parameters and transmits them synchronously to the deep reinforcement learning models.

9. The method for path planning of a ground robot based on deep reinforcement learning according to claim 1, characterized in that, After training the deep reinforcement learning model, robot dynamics information and real-time elevation map are input into the deep reinforcement learning model that stores the optimal weight parameters to obtain the real-time optimal path planning strategy, thereby realizing real-time path planning for the ground robot.

Images