A path planning method for park logistics vehicles based on map-free navigation

Abstract

The present invention discloses a path planning method for a park logistics vehicle based on mapless navigation, comprising: 1. constructing a simulation scene of the park in Gazebo and building a vehicle model; 2. defining state parameters, action parameters and reward functions of a deep reinforcement learning method for mapless navigation; 3. constructing a neural network model, initializing the neural network parameters, training the neural network model, and obtaining an optimal network neural network model; 4. obtaining the current state parameter St of the vehicle, and outputting the speed and steering angle of the vehicle according to the optimal neural network; 5. modeling the difference between the simulation environment and the real world, and learning the mapping from the vehicle in the simulation environment to the vehicle in the real environment. 6. combining the trained model with the simulation environment to learn a strategy for transferring the training model of the vehicle to the target environment. Based on mapless navigation technology, the present invention improves the navigation effect of the logistics vehicle through the perception information of the on-board terminal and the sensor.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a path planning method for a park logistics vehicle based on map-free navigation. Background Art

[0002] 1) Currently, most industrial park logistics and transportation are carried out manually. However, there is a huge gap between the number of working population and the demand for labor, and the difficulty in finding workers for logistics and distribution is difficult to alleviate.

[0003] 2) Mapless navigation can find a path for the car from the starting point to the target location. The path should be as short as possible, the smoothness of the path should meet the dynamic characteristics of the car, and the safety of the path should ensure that the car will not collide and the goods can be safely delivered to the target point.

[0004] 3) Currently, the realization of navigation functions in park warehouses mainly relies on SLAM mapping, which actively builds maps to obtain environmental maps and simultaneously determine the position of the vehicle. However, traditional SLAM mapping mainly focuses on using geometric techniques to build maps, and navigation itself is rarely considered. The representations constructed by SLAM systems are usually not suitable for traditional motion planning methods. In addition, during the mapping process, dynamic objects in the park environment will have dynamic feature points, resulting in poor mapping robustness and map drift. Mapless navigation based on reinforcement learning does not rely on maps and has the ability to learn during the interaction between the vehicle and the environment. It has stronger adaptability in complex dynamic environments such as logistics parks. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a path planning method for a park logistics vehicle based on map-free navigation, so as to more effectively learn the vehicle's path planning ability without a map in complex and changeable real environments such as logistics parks where it is difficult for logistics vehicles to build maps. This can effectively shorten the time required for the vehicle to be trained in a real environment, and improve the safety of the vehicle's path planning in a real environment, thereby providing a reference for the improvement and optimization of map-free navigation.

[0006] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0007] The path planning method for a park logistics vehicle based on map-free navigation of the present invention is characterized in that it includes the following steps:

[0008] Step 1: Design the dynamic model of the car in the simulation environment according to the real environment car, and set the initial parameters of the car in the simulation environment, including: maximum driving speed v max and the maximum rotation angle θ max; According to the spatial state of obstacles in the real environment, set the same spatial state of obstacles in the simulation environment;

[0009] Step 2: Define the state parameter set Ss of the car in the simulation environment = {s1 s ,s2 s ...,s i s ,...,sms},s i s is the normalized state of the car in the simulation environment at step i, including: the minimum distance between the car and the obstacle at step i, the distance di from the car's position at step i to the target point, and the coordinates of the car at step i; i = 1, 2, …, m, where m is the total number of steps;

[0010] Define the action parameter set A of the car in the simulation environment = {a1, a2..., a i ,...,a m}, a i is the state of the car at step i, and a i ={v i ,θ i}, where v i is the speed of the car at step i, θ i is the steering angle of the car at step i;

[0011] Step 3: According to the action amount a of the car in step i i , generate the minimum distance between the car and the obstacle at step i, and the distance d from the car's position at step i to the target point i And the coordinates of the car at step i;

[0012] Step 4: Set the reward function R of the car at step i i ;

[0013] Step 5: Construct a strategy-evaluation network π and train it using the PPO algorithm to obtain a local optimal strategy-evaluation model π.

[0014] Step 6: Build an LSTM model φ based on the difference between the car in the simulation environment and the real environment, and train the LSTM model φ. Then, combine the trained LSTM model φ with the real environment to train the strategy-evaluation network to obtain the global optimal strategy-evaluation network π. * .

[0015] Step 7: The global optimal strategy-evaluation model π *Deploy it to the control module of the car in the real environment, and after obtaining the global optimal minimum distance between the car and the obstacle at step i, the global optimal distance from the car's position at step i to the target point, and the global optimal coordinates of the car according to the process in step 3, input the global optimal strategy-evaluation model π * , thereby outputting the global optimal action of the car in the i+1th step and moving.

[0016] The path planning method for a park logistics vehicle based on map-free navigation according to the present invention is also characterized in that step 4 is to construct a reward function R using formula (1) i :

[0017] R i =ω1R vi +ω2R si +ω3R ci +ω4R ai (2)

[0018] In formula (2), R vi Represents the speed reward function, and is obtained from formula (3), R si Represents the steering reward function, and is obtained from formula (4), R ci represents the collision reward function, and is obtained from formula (5), R ai represents the reward function close to the target, and is obtained by formula (7); ω1, ω2, ω3, ω4 represent the weights of the four rewards;

[0019]

[0020] In formula (3), r represents the speed reward coefficient, v max Indicates the maximum speed of the car, v min Indicates the minimum speed of the car;

[0021]

[0022] In formula (4), τr eward Indicates the positive reward value when the car does not change direction obviously, τ punish It represents the penalty value for the car to change direction obviously, and θ0 represents the standard for judging whether the car is turning obviously.

[0023]

[0024] In formula (5), D min Indicates the minimum collision warning circle, v max Indicates the maximum driving speed, D max Indicates the maximum collision warning circle, d iminrepresents the minimum distance to the obstacle, σ1 represents the penalty value for the car to collide, σ2 represents the penalty parameter for the car to collide, and:

[0025]

[0026] In formula (6), d max represents the maximum collision threshold, d min represents the minimum collision threshold; Δt is the time interval;

[0027]

[0028] In formula (7), c d Indicates the threshold value of the car reaching the target point, c r Represents the distance change reward coefficient, c i Indicates the penalty for each additional time step, d i is the distance between the current car and the target point, c reward Indicates the reward value given when the car reaches the target point.

[0029] The step 5 comprises:

[0030] Step 5.1: The strategy-evaluation network π includes: an evaluation network and a strategy network;

[0031] The evaluation network is composed of two fully connected layers and an output layer, and the activation functions between all fully connected layers are ReLU functions; let the parameters of the evaluation network be The input of the evaluation network is the normalized state quantity, and the output is the current state value;

[0032] The strategy network includes: a neural network for the output mean part and a neural network for the output variance part; wherein the neural network for the output mean part includes: two fully connected layers and an output layer, and the output layer activation function is a tanh function; the neural network for the output variance part includes: two fully connected layers and an output layer, and the output layer activation function is a softplus activation function; the activation function between all fully connected layers is a Relu function;

[0033] Let the parameters of the policy network be θ, the input of the policy network is the normalized state quantity, and the output is the current action quantity a~N(ε,std); ε and std represent the mean and variance respectively; N represents Gaussian distribution, and ~ represents obedience;

[0034] Define the training strategy of the policy network as π θ , define the environment interaction parameter of the policy network as θ', and the environment interaction strategy of the policy network as π θ' ;

[0035] Step 5.2: Normalized state s of step i i Input the strategy network for processing and output the action amount a of step i i and the reward R for step i i ;

[0036] The action amount a of step i i The expected trajectory of step i is obtained by processing. After the vehicle dynamics model executes the expected trajectory of step i, the normalized state quantity s of step i+1 is obtained. i+1 ;

[0037] (s i ,a i ,s i+1 ,R i ) is stored in the experience pool as the i-th sample;

[0038] Step 5.3: After assigning i+1 to i, return to step 5.2 until the number of samples in the experience pool reaches the upper limit n;

[0039] Step 5.4: Draw any i-th sample from the experience pool and use the advantage function estimator shown in formula (8) to calculate the estimated value of the i-th step

[0040]

[0041] In formula (8), V φ (s i ) is to convert the The state value obtained after inputting the value network, V φ (s i+1 ) is the number of samples in the i-th The state value obtained after inputting the value network; γ is the reward discount coefficient;

[0042] Step 5.5: Define the state The strategy π under θ Select action a i The probability of π(a i |s i ,θ i ), in the state quantity The strategy π under θ' Select action a i The probability of π(a i |s i ,θ' i ), define π(a i |s i ,θ i ) and π(a i |s i, θ' i ) has a ratio of r(θ i ); θ i represents the network parameters of the policy network at the i-th step;

[0043] Step 5.6: Use Equation (8) to obtain the network parameters θ of the policy network at the (i + 1)-th step i+1 :

[0044]

[0045] In Equation (9), ▽ θ is the gradient operator of the policy network parameter θ, clip(r(θ i ), 1 - ε, 1 + ε) is the range limiting function, which limits r(θ i ) within the range of 1 - ε to 1 + ε, and ε is the set threshold, is the expectation;

[0046] s Step 5.7: Use Equation (9) to obtain the network parameters of the value network at the (i + 1)-th step

[0047] s

[0048] In Equation (10), is the gradient operator of the value network parameter ;

[0049] Step 5.8: After assigning i + 1 to i, if i < m, then return to Step 5.2 and execute sequentially, otherwise, it means obtaining the network parameters θ m and and form the optimal policy - evaluation network model π.

[0050] The said Step 6 includes:

[0051] Step 6.1: Initialize i = 1;

[0052] Use the Markov decision process to represent the simulation environment D s s = <S s , A, p[[ID=NO]] s , r s >, and the real environment D t = <S t , A, p t , r t >, where r[[ID=NO]] s represents the reward function of the simulation environment, r[[ID=NO]] t represents the reward function of the real environment, and r[[ID=NO]] s = r [[ID=NO]] t = R i , p sWith p t Represent the state transition probability distribution of the simulation environment and the real environment respectively;

[0053] Step 6.2: Sample the state of the car in the real environment at step i-1 s i-1 t ~p t (s i-1 ), where s i-1 t Indicates the state of the car in the real environment at step i-1, p t (s i-1 ) represents the state s of the car in the real environment at step i-1 i-1 t The state transition probability distribution of

[0054] Step 6.3: Set the state s of the car in the simulation environment at step i-1 from the state s of the car in the real environment i-1 t start;

[0055] Give the car a random and safe behavior strategy μ so that the car follows the strategy a i ~μ(s i t ) to perform the movement of step i and collect the state s of the car in step i in the real environment i t And the state s of the car in the simulation environment at step i i s , thus obtaining the i-th experience (s i ,a i ,s i+1 s ,s i+1 t );

[0056] Step 6.4: After assigning i+1 to i, return to step 6.3 and execute sequentially until i=T, thus obtaining a trajectory τ=(s0 s ,s0 t ,s1 s ,s1 t ...s T-2 s ,a T ,s T-1 s ,s T-1 t );

[0057] Step 6.5: Based on the trajectory τ, use the Adam algorithm to perform maximum likelihood training on the LSTM model φ to obtain the trained LSTM model φ, thereby predicting the estimated state of the car in the real environment at the i+1 step

[0058] The LSTM model φ uses formula (1) to predict the state s of the car in the real environment at the i+1 step i+1 t :

[0059] φ(s i t ,a i ,h,s i+1 s )=s i+1 t -s i+1 s (1)

[0060] In formula (1), h is the hidden state of the LSTM model φ;

[0061] Step 6.6: According to the local optimal strategy-evaluation model π, obtain the action sample a of the car in the simulation environment at step i i ~π(s i s ) and execute, thereby obtaining the reward R of the car in the simulation environment at step i i And the state s at step i+1 i+1 s ; thus forming a sample Stored in the experience pool, used to update the local optimal strategy-evaluation model π to obtain the updated local optimal strategy-evaluation model π';

[0062] Step 6.7: Assign π' to π, and set the state s of the car in the simulation environment at step i+1 to i+1 s Compared with the state estimation of the car in the real environment at step i+1 After the same, return to step 6.3 and execute sequentially until i = T, thus obtaining the global optimal strategy-evaluation model π * .

[0063] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the path planning method, and the processor is configured to execute the program stored in the memory.

[0064] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the path planning method when the computer program is executed by a processor.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. Due to factors such as changes in cargo and movement of personnel in the logistics park, the present invention adopts map-free navigation technology and uses reinforcement learning algorithms for navigation, which can significantly reduce the collision rate of mobile robots during navigation and improve the navigation success rate, thereby improving the safety performance of reinforcement learning map-free navigation.

[0067] 2. For dynamic environments such as parks, the car of the present invention is trained in a simple environment and then migrated to a complex dynamic environment. In the absence of a map, it can effectively use the obstacle information fed back by the environment to make better navigation decisions. When encountering dynamic obstacles, it can also predictively avoid them, showing strong generalization capabilities.

[0068] 3. There is still the problem of low sampling efficiency when the car is directly trained and transferred in the real environment. The present invention adopts the neural enhanced simulation (NAS) learning control strategy to model the difference between the simulation environment and the real environment, learn the mapping from the simulation environment to the real environment, and more effectively utilize the strategy learned in the simulation environment, thereby reducing the training time of the car in reality.

[0069] 4. There are also safety issues with allowing the car to perform training transfer directly in a real environment. Since reinforcement learning requires the agent to conduct trial and error through large-scale random sampling in the environment, its actions at certain moments may damage the car itself, the surrounding environment, or even living things. This invention uses sim2real technology to simulate the real environment in a simulation environment to improve the safety of the car during map-free navigation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the PPO algorithm framework implemented in the present invention;

[0071] Figure 2 This is a schematic diagram of the algorithm training process in an embodiment of the present invention;

[0072] Figure 3 This is a diagram of the Sim-to-Real training model in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In this embodiment, a path planning method for a park logistics vehicle based on map-free navigation includes the following steps:

[0074] Step 1: Design the dynamic model of the car in the simulation environment according to the real environment car, and set the initial parameters of the car in the simulation environment, including: maximum driving speed v maxand the maximum rotation angle θ max ; According to the spatial state of obstacles in the real environment, set the same spatial state of obstacles in the simulation environment.

[0075] Step 2: Define the state parameter set S of the car in the simulation environment s ={s1 s ,s2 s ...,s i s ,...,s m s}, s i s is the normalized state of the car in the simulation environment at step i, including: the minimum distance between the car and the obstacle at step i, the distance d from the position of the car at step i to the target point i , the coordinates of the car at step i; i = 1, 2, …, m, where m is the total number of steps;

[0076] Define the action parameter set A of the car in the simulation environment = {a1, a2..., a i ,...,a m}, a i is the state of the car at step i, and a i ={v i ,θ i}, where v i is the speed of the car at step i, θ i is the steering angle of the car at step i.

[0077] Step 3: According to the action amount a of the car in step i i , generate the minimum distance between the car and the obstacle at step i, and the distance d from the car's position at step i to the target point i And the coordinates of the car at step i.

[0078] Step 4: Set the reward function R of the car at step i i ;

[0079] Step 4.1: Formula (1) constructs the reward function R i :

[0080] R i =ω1R vi +ω2R si +ω3R ci +ω4R ai (1)

[0081] In formula (1), R vi Represents the speed reward function, and is obtained from formula (2), R si Represents the steering reward function, and is obtained from formula (3), Rci Represents the collision reward function, and is obtained from formula (4), R ai represents the reward function close to the target, and is obtained by formula (6); ω1, ω2, ω3, ω4 represent the weights of the four rewards;

[0082]

[0083] In formula (2), r represents the speed reward coefficient, v max Indicates the maximum speed of the car, v min Indicates the minimum speed of the car;

[0084]

[0085] In formula (3), τ reward Indicates the positive reward value when the car does not change direction obviously, τ punish represents the penalty value for the car to change direction obviously, and θ0 represents the standard for judging whether the car has turned obviously;

[0086]

[0087] In formula (4), D min Indicates the minimum collision warning circle, v max Indicates the maximum driving speed, D max Indicates the maximum collision warning circle, d imin represents the minimum distance to the obstacle, σ1 represents the penalty value for the car to collide, σ2 represents the penalty parameter for the car to collide, and:

[0088]

[0089] In formula (5), d max represents the maximum collision threshold, d min represents the minimum collision threshold; Δt is the time difference between training steps;

[0090]

[0091] In formula (6), c d Indicates the threshold value of the car reaching the target point, c r Represents the distance change reward coefficient, c i Indicates the penalty for each additional time step, d i is the distance between the current car and the target point, c reward Indicates the reward value given when the car reaches the target point.

[0092] Step 5: Construct a strategy-evaluation network π and train the strategy-evaluation network using the PPO algorithm to obtain a local optimal strategy-evaluation model π. The schematic diagram of the PPO algorithm framework implemented in the present invention is as follows: Figure 1 shown.

[0093] Step 5.1: The policy-criticism network π includes: the criticism network and the policy network;

[0094] The evaluation network is composed of two fully connected layers and an output layer, and the activation functions between all fully connected layers are ReLU functions; let the parameters of the evaluation network be The input of the evaluation network is the normalized state quantity, and the output is the current state value;

[0095] The strategy network consists of two neural networks for the mean output and the variance output. The mean output neural network consists of two fully connected layers and an output layer, with the output layer activation function being the tanh function. The variance output neural network consists of two fully connected layers and an output layer, with the output layer activation function being the softplus activation function. The activation function between all fully connected layers is the Relu function.

[0096] Let the parameters of the policy network be θ, the input of the policy network is the normalized state quantity, and the output is the current action quantity a~N(ε,std); ε and std represent the mean and variance respectively; N represents Gaussian distribution, and ~ represents obedience;

[0097] Define the training strategy of the policy network as π θ , define the environment interaction parameter of the policy network as θ', and the environment interaction strategy of the policy network as π θ' .

[0098] Step 5.2: Normalized state s of step i i Input the policy network for processing and output the action amount a of step i i and the reward R for step i i ;

[0099] The action amount a of step i i The expected trajectory of step i is obtained by processing. After the vehicle dynamics model executes the expected trajectory of step i, the normalized state quantity s of step i+1 is obtained. i+1 ;

[0100] (s i ,a i ,s i+1 ,R i ) is stored in the experience pool as the i-th sample.

[0101] Step 5.3: After assigning i+1 to i, return to step 5.2 until the number of samples in the experience pool reaches the upper limit n;

[0102] Step 5.4: Draw any i-th sample from the experience pool and use the advantage function estimator shown in formula (8) to calculate the estimated value of the i-th step

[0103]

[0104] In formula (7), V φ (s i ) is to convert the The state value obtained after inputting the value network, V φ (s i+1 ) is the number of samples in the i-th The state value obtained after inputting the value network; γ is the reward discount coefficient.

[0105] Step 5.5: Define the state The strategy π under θ Select action a i The probability of π(a i |s i ,θ i ), in the state quantity The strategy π under θ' Select action a i The probability of π(a i |s i ,θ' i ), define π(a i |s i ,θ i ) and π(a i |s i ,θ' i ) is r(θ i );θ i Represents the network parameters of the policy network at step i;

[0106] Step 5.6: Use formula (8) to get the network parameters θ of the policy network at step i+1 i+1 :

[0107]

[0108] In formula (8), θ is the gradient operator of the policy network parameter θ, clip(r(θ i ),1-ε,1+ε) is the range limiting function, which is r(θ i ) is limited to the range of 1-ε to 1+ε, where ε is the set threshold. It is an expectation.

[0109] Step 5.7: Obtain the network parameters of the value network at the (i + 1)-th step using Equation (9).

[0110]

[0111] In Equation (9), is the gradient operator of the value network parameters .

[0112] Step 5.8: After assigning i + 1 to i, if i < m, then return to Step 5.2 and execute sequentially; otherwise, it means obtaining the network parameters θ at the m-th step. m and and constitute the optimal policy-evaluation network model π. The training process of this policy-evaluation network is as Figure 2 shown.

[0113] Step 6: Build an LSTM model φ based on the differences between the simulation environment and the real environment of the trolley, train the LSTM model φ, and combine the trained LSTM model φ with the real environment to train the policy-evaluation network to obtain the global optimal policy-evaluation network.

[0114] LSTM model φ based on the differences between the simulation environment and the real environment of the trolley, train the LSTM model φ, and combine the trained LSTM model φ with the real environment to train the policy-evaluation network to obtain the global optimal policy-evaluation network. Step 6.1: Initialize i = 1;

[0115] Use the Markov decision process to represent the simulation environment D s =<S s , A, p s , r s > and the real environment D t =<S t , A, p t , r t . Here, r s represents the reward function of the simulation environment, r t represents the reward function of the real environment, and r [[ID=ST]] s = r t = R i , p s and p t represent the state transition probability distributions of the simulation environment and the real environment respectively.

[0116] Step 6.2: Sample the state of the trolley at the (i - 1)-th step in the real environment s i-1 t ~p [[ID=ST]] t (s i-1 ), where s i-1 t represents the state of the trolley at the (i - 1)-th step in the real environment, p t (si-1 ) represents the state s of the car in the real environment at step i-1 i-1 t The state transition probability distribution of .

[0117] Step 6.3: Set the state s of the car in the simulation environment at step i-1 from the state s of the car in the real environment i-1 t start;

[0118] Give the car a random and safe behavior strategy μ so that the car follows the strategy a i ~μ(s i t ) to perform the movement of step i and collect the state s of the car in step i in the real environment i t And the state s of the car in the simulation environment at step i i s , thus obtaining the i-th experience (s i ,a i ,s i+1 s ,s i+1 t );

[0119] Step 6.4: After assigning i+1 to i, return to step 6.3 and execute sequentially until i=T, thus obtaining a trajectory τ=(s0 s ,s0 t ,s1 s ,s1 t ...s T-2 s ,a T ,s T-1 s ,s T-1 t );

[0120] Step 6.5: Based on the trajectory τ, use the Adam algorithm to perform maximum likelihood training on the LSTM model φ to obtain the trained LSTM model φ, thereby predicting the estimated state of the car in the real environment at the i+1 step

[0121] The LSTM model φ uses formula (1) to predict the state s of the car in the real environment at the i+1 step i+1 t :

[0122] φ(s i t ,a i ,h,s i+1 s )=s i+1t -s i+1 s (10)

[0123] In formula (10), h is the hidden state of the LSTM model φ.

[0124] Step 6.6: According to the local optimal strategy-evaluation model π, obtain the action sample a of the car in the simulation environment at step i i ~π(s i s ) and execute, thereby obtaining the reward R of the car in the simulation environment at step i i And the state s at step i+1 i+1 s ; thus forming a sample It is stored in the experience pool and used to update the local optimal strategy-evaluation model π to obtain the updated local optimal strategy-evaluation model π':

[0125] Step 6.7: Assign π' to π, and set the state s of the car in the simulation environment at step i+1 to i+1 s Compared with the state estimation of the car in the real environment at step i+1 After the same, return to step 6.3 and execute sequentially until i = T, thus obtaining the global optimal strategy-evaluation model π * ;

[0126] Figure 3 The left side shows the method of training the forward dynamics model by collecting the state differences when running the same action on the simulation and the real car. Figure 3 The right side shows that once the forward model is learned, it can be applied to the simulated state to obtain the corresponding real state. This correction model is time-dependent and uses LSTM, and the policy learning algorithm can only be used through the real car state.

[0127] Step 7: The global optimal strategy-evaluation model π * Deploy it to the control module of the car in the real environment, and after obtaining the global optimal minimum distance between the car and the obstacle at step i, the global optimal distance from the car's position at step i to the target point, and the global optimal coordinates of the car according to the process in step 3, input the global optimal strategy-evaluation model π * , thereby outputting the global optimal action of the car in the i+1th step and moving.

[0128] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0129] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. A path planning method for a park logistics vehicle based on map-free navigation, characterized in that: The following steps are involved: Step 1: Design the dynamic model of the car in the simulation environment according to the real environment car, and set the initial parameters of the car in the simulation environment, including: maximum driving speed and maximum turning angle ; According to the spatial state of obstacles in the real environment, set the same spatial state of obstacles in the simulation environment; Step 2: Define the state parameter set of the car in the simulation environment , is the normalized state of the car in the simulation environment at step i, including: the minimum distance between the car and the obstacle at step i, the distance from the car's position at step i to the target point , the coordinates of the car at step i; , m is the total number of steps; Define the action parameter set of the car in the simulation environment , is the state of the car at step i, and ,in, is the speed of the car at step i, is the steering angle of the car at step i; Step 3: According to the amount of movement of the car in step i , generate the minimum distance between the car and the obstacle at step i, and the distance from the car at step i to the target point And the coordinates of the car at step i; Step 4: Speed ​​reward function , turning to the reward function , collision reward function , close to the target reward function Together we set the reward function for the car at step i ; Step 5: Build a strategy-evaluation network , and use the PPO algorithm to train the strategy-evaluation network to obtain the local optimal strategy-evaluation model ; Step 6: Build an LSTM model based on the differences between the car in the simulation environment and the real environment , and the LSTM model Train the LSTM model Combined with the real environment, it is used to train the strategy-evaluation network to obtain the global optimal strategy-evaluation network ; Step 7: Global optimal strategy-evaluation model Deployed to the control module of the real-world car, and after obtaining the global optimal minimum distance between the car and the obstacle at step i, the global optimal distance from the car's position at step i to the target point, and the global optimal coordinates of the car according to the process of step 3, input the global optimal strategy-evaluation model , thereby outputting the global optimal action of the car in the i+1th step and moving.

2. A path planning method for a park logistics vehicle based on map-free navigation according to claim 1, characterized in that: The step 4 is to construct the reward function using formula (2) : (2) In formula (2), represents the speed reward function, and is obtained from formula (3), represents the steering reward function, and is obtained from formula (4), represents the collision reward function, and is obtained from Equation (5), represents the close target reward function and is obtained by formula (7); , , , Indicates the weights of the four rewards; (3) In formula (3), represents the speed bonus coefficient, Indicates the maximum speed of the car, Indicates the minimum speed of the car; (4) In formula (4), Indicates the positive reward value when the car does not change direction obviously. Indicates the penalty value for the car to change direction significantly. Indicates the standard for judging whether the car is turning obviously; (5) In formula (5), Indicates the minimum collision warning circle, Indicates the maximum driving speed. Indicates the maximum collision warning circle, Indicates the minimum distance to the obstacle. Indicates the penalty value for a car collision. It represents the penalty parameter for the car to collide, and has: (6) In formula (6), represents the maximum collision threshold, Indicates the minimum collision threshold; For the time interval; (7) In formula (7), Indicates the threshold value near the target point where the car reaches. Represents the distance change reward coefficient, Indicates the penalty for each extra time step. is the distance between the current car and the target point, Indicates the reward value given when the car reaches the target point.

3. The path planning method for a park logistics vehicle based on map-free navigation according to claim 1 is characterized in that: The step 5 comprises: Step 5.1: The policy-criticism network Includes: evaluation network and strategy network; The evaluation network is composed of two fully connected layers and an output layer, and the activation functions between all fully connected layers are ReLU functions; let the parameters of the evaluation network be , the input of the evaluation network is the normalized state quantity, and the output is the current state value; The strategy network includes: a neural network for the output mean part and a neural network for the output variance part; wherein the neural network for the output mean part includes: two fully connected layers and an output layer, and the output layer activation function is a tanh function; the neural network for the output variance part includes: two fully connected layers and an output layer, and the output layer activation function is a softplus activation function; the activation function between all fully connected layers is a Relu function; Let the parameters of the policy network be , the input of the policy network is the normalized state quantity, and the output is the current action quantity a~N(ε, std); ε and std represent the mean and variance respectively; N represents Gaussian distribution, and ~ represents obedience; Define the training strategy of the policy network as , define the environment interaction parameters of the policy network as , the environmental interaction strategy of the policy network is ; Step 5.2: Normalize the state quantity of step i Input the strategy network for processing and output the action amount of step i and the reward for step i ; The action amount of step i The expected trajectory of step i is obtained by processing. After the dynamic model executes the expected trajectory of step i, the normalized state quantity of step i+1 is obtained. ; Will As the first i The samples are stored in the experience pool; Step 5.3: After assigning i+1 to i, return to step 5.2 until the number of samples in the experience pool reaches the upper limit of capacity. until; Step 5.4: Draw any i samples, and thus the advantage function estimator shown in formula (8) is used to calculate the i Estimated value of step : (8) In formula (8), To convert the i-th step The state value obtained after entering the value network, To be the first i In the sample The state value obtained after inputting the value network; is the reward discount coefficient; Step 5.5: Define the state The strategy below Select Action The probability of , in the state quantity The strategy below Select Action The probability of ,definition and The ratio is ; Represents the network parameters of the policy network at step i; Step 5.6: Use formula (8) to get the network parameters of the policy network at step i+1 : (9) In formula (9), is the policy network parameter The gradient operator, is the range limiting function, which is to Restricted to arrive Within the range, is the set threshold, For expectations; Step 5.7: Use formula (9) to get the network parameters of the value network at step i+1 : (10) In formula (10), is the value network parameter The gradient operator of Step 5.8: After assigning i+1 to i, if i< If , then return to step 5.2 to execute sequentially, otherwise, it means that the Network parameters of the step and And form the optimal strategy-evaluation network model .

4. A path planning method for a park logistics vehicle based on map-free navigation according to claim 3, characterized in that: The step 6 comprises: Step 6.1: Initialize i=1; The Markov decision process will be used to represent the simulation environment and the real environment ,in, represents the reward function of the simulation environment, represents the reward function of the real environment, and = , and Represent the state transition probability distribution of the simulation environment and the real environment respectively; Step 6.2: Sample the state of the car in the real environment at step i-1 ,in, Indicates the state of the car in the real environment at step i-1, Indicates the state of the car in the real environment at step i-1 The state transition probability distribution of Step 6.3: Set the car in the simulation environment to the state of the car in the real environment at step i-1 start; Give the car a random and safe behavior strategy , so that the car follows the strategy Perform the i-th step movement and collect the state of the car in the i-th step in the real environment And the state of the car in the simulation environment at step i , thus obtaining the i-th experience ; Step 6.4: After assigning i+1 to i, return to step 6.3 and execute sequentially until i=T, thus obtaining a trajectory of length T. ; Step 6.5: Trajectory-based , using the Adam algorithm to LSTM model Perform maximum likelihood training to obtain the trained LSTM model , thereby predicting the estimated state of the car in the real environment at the i+1 step ; LSTM model Use formula (1) to predict the state of the car in the i+1 step in the real environment : (1) In formula (1), h is the LSTM model The hidden state of Step 6.6: Based on the local optimal strategy - evaluation model , get the action sample of the car in the simulation environment at step i And execute, so as to obtain the reward of the car in the simulation environment at step i And the state of step i+1 ; thus forming a sample Stored in the experience pool for local optimal strategy-evaluation model Update to obtain the updated local optimal strategy-evaluation model '; Step 6.7: 'Assign to , the state of the car in the simulation environment at step i+1 Compared with the state estimation of the car in the real environment at step i+1 After the same, return to step 6.3 and execute sequentially until i=T, thus obtaining the global optimal strategy-evaluation model .

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the path planning method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the path planning method according to any one of claims 1 to 4 are executed.

Images