Unmanned vehicle autonomous path planning method and device, medium and product

Abstract

The invention discloses an unmanned vehicle autonomous path planning method and device, a medium and a product, and relates to the technical field of automatic driving, and the method comprises the steps: constructing a state fusion correction framework; the state fusion correction framework comprises an action prediction network, a state prediction network and a state fusion module; constructing a strategy network based on a deep reinforcement learning algorithm; the strategy network is used for outputting an executed action according to the fusion state; determining a path planning model according to the state fusion correction framework and the strategy network; determining the state of the to-be-planned unmanned vehicle according to the target position, the unmanned vehicle position and the radar point cloud information; and according to the state of the to-be-planned unmanned vehicle, adopting the trained path planning model to obtain an executed action. According to the invention, the path planning capability of the unmanned vehicle can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous path planning method, device, medium and product for an unmanned vehicle. Background Art

[0002] Unmanned vehicles are an important application of outdoor mobile robots in the field of transportation. As one of the most widely used unmanned systems, unmanned vehicles can complete most ground tasks, including target search, disaster relief, logistics and transportation, etc. In most ground tasks, unmanned vehicles are required to have the ability to plan paths. Therefore, path planning is a basic function that unmanned vehicles must have to complete complex tasks, and it is also one of the most important capabilities.

[0003] The A* algorithm is a widely used heuristic algorithm in the field of path planning. Many researchers have improved the above algorithm based on the kinematic characteristics of unmanned vehicles. However, the actual mission environment of unmanned vehicles is complex and changeable, and these traditional algorithms are difficult to accurately describe their dynamic models. In addition, when the complexity of the environment increases, such algorithms also require more computing resources.

[0004] Therefore, the path planning of unmanned vehicles faces problems such as difficulty in kinematic and dynamic modeling and dynamic changes in controller parameters. The development of reinforcement learning technology has opened up a new path to solve the path planning problem of unmanned vehicles. Reinforcement learning does not require kinematic and dynamic modeling of unmanned vehicles. It can evaluate the execution actions of the intelligent agent through continuous interaction with the environment and the reward function, and then gradually learn a complete control strategy. However, in the reinforcement learning framework, the state transition of the intelligent agent conforms to the Markov property, which is a prerequisite for ensuring the convergence of the training process. However, in real control tasks, there are many factors that cause the state transition of the intelligent agent to not satisfy the Markov property; on the one hand, the strategy generates the execution action itself, which requires a certain amount of computing time. When the strategy is more complex or the environment is more complex, it often requires more computing resources and computing time to generate the execution action. On the other hand, it also takes a certain amount of time for the unmanned vehicle to respond to the action after receiving the action instruction, which leads to the state transition of the unmanned vehicle not conforming to the Markov property, further resulting in slow network convergence and failure to carry out the path planning of the unmanned vehicle in time.

[0005] Therefore, in order to improve the path planning capability of unmanned vehicles, it is urgent to provide an unmanned vehicle autonomous path planning method or device. Summary of the invention

[0006] The purpose of this application is to provide an unmanned vehicle autonomous path planning method, equipment, medium and product, which can improve the path planning capability of the unmanned vehicle.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides an autonomous path planning method for an unmanned vehicle, the autonomous path planning method for an unmanned vehicle comprising:

[0009] Construct a state fusion correction framework; the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to output a predicted action according to the state at time t; the state prediction network is used to output a predicted state according to the state at time t and the predicted action; the state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state; the state includes: the distance between the unmanned vehicle and the target, the position of the target position in the unmanned vehicle and the radar point cloud information;

[0010] Based on the deep reinforcement learning algorithm, a policy network is constructed; the policy network is used to perform actions according to the fusion state output;

[0011] Determine a path planning model based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output;

[0012] Determine the status of the unmanned vehicle to be planned based on the target location, the unmanned vehicle location, and the radar point cloud information;

[0013] According to the state of the unmanned vehicle to be planned, the trained path planning model is used to obtain the execution action.

[0014] Optionally, the reward function r of the policy network t It is expressed as:

[0015]

[0016] Among them, r d represents the distance reward, r d =tand(d t-1 -d t ), the tanh function is used for normalization, d t represents the distance between the unmanned vehicle and the target at time t, d t-1 represents the distance between the unmanned vehicle and the target at time t-1, x g Indicates the horizontal coordinate of the target position, y g The ordinate of the target position. The horizontal coordinate representing the position of the unmanned vehicle, The ordinate of the unmanned vehicle’s position, r v Indicates line speed bonus, v t represents the linear velocity, v min Indicates the minimum linear velocity, v maxIndicates the maximum linear velocity, r ω represents the angular velocity reward, ω t represents the angular velocity, ω max represents the maximum angular velocity, r c represents the state error reward, r c = -tanh(|s t+1 -s′ t+1 |),s t+1 represents the state at time t+1, s′ t+1 Represents the predicted state at time t+1.

[0017] Optionally, the objective function J(θ) of the policy network is expressed as:

[0018]

[0019] Among them, p θ (a t |s t ) represents the action output probability of the policy network, represents the action output probability of the policy network before updating, θ represents the parameters of the policy network, τ represents a batch of samples, b represents the expectation of reward, γ is a hyperparameter, and k represents the number of times the model is updated for each batch of samples. represents the intermediate parameter, R(τ)=r1+γr2+γ 2 r3+...+γ n-1 r n , clip() represents the program function, ε represents the hyperparameter, R(τ) represents the total reward value obtained by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle, r1, r2, r3 and r n It represents the reward value obtained by the unmanned vehicle when performing the corresponding action, and n represents the number of actions performed by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle.

[0020] Optionally, the training process of the path planning model is:

[0021] Create ground training and validation simulation environments;

[0022] Initialize the network parameters in the policy network and state fusion correction framework;

[0023] Get the original state of the unmanned vehicle at time t;

[0024] The original state at time t is input into the state fusion correction framework, and the fusion state is obtained through the action prediction network, state prediction network and state fusion module;

[0025] The fusion state is input into the policy network, the policy network outputs the executed action, the unmanned vehicle performs the corresponding action and returns the reward value;

[0026] Obtain the fusion state at time t+1, and store the fusion state at time t, the executed action, the reward value, and the fusion state at time t+1 in the experience replay pool; until a set number of samples are obtained, update the parameters of the path planning model;

[0027] The training is completed until the average reward value of the driverless car reaches the set threshold.

[0028] Optionally, create a ground-based training and validation simulation environment, including:

[0029] Use Gazebo under Linux system to create a ground training and verification simulation environment.

[0030] Optionally, determining the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle and the radar point cloud information specifically includes:

[0031] Determine the distance between the unmanned vehicle and the target based on the target position and the unmanned vehicle position;

[0032] Using the formula Determine the position of the target position at time t in the direction of the unmanned vehicle ρ t , x g Indicates the horizontal coordinate of the target position, y g The ordinate of the target position. The horizontal coordinate representing the position of the unmanned vehicle, The vertical coordinate representing the position of the unmanned vehicle.

[0033] In a second aspect, the present application provides an unmanned vehicle autonomous path planning device, the unmanned vehicle autonomous path planning device comprising:

[0034] The state fusion correction framework construction unit is used to construct the state fusion correction framework; the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to output a predicted action according to the state at time t; the state prediction network is used to output a predicted state according to the state at time t and the predicted action; the state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state; the state includes: the distance between the unmanned vehicle and the target, the position of the target position in the unmanned vehicle and the radar point cloud information;

[0035] A policy network construction unit, used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to perform actions according to the fusion state output;

[0036] A path planning model determination unit, used to determine a path planning model according to the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output;

[0037] The state determination unit of the unmanned vehicle is used to determine the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle and the radar point cloud information;

[0038] The path planning unit is used to obtain the execution action according to the state of the unmanned vehicle to be planned using the trained path planning model.

[0039] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the unmanned vehicle autonomous path planning method.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unmanned vehicle autonomous path planning method.

[0041] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the unmanned vehicle autonomous path planning method when executed by a processor.

[0042] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0043] The present application provides an autonomous path planning method, device, medium and product for an unmanned vehicle. A policy network is constructed based on a deep reinforcement learning (Proximal Policy Optimization, PPO) algorithm, which can output continuous action values, ensure the continuity of state mapping actions, and improve the path planning effect of the unmanned vehicle. The action prediction network in the constructed state fusion correction framework predicts the action through the original state at the current moment, provides a basis for the state prediction network, and ensures the accuracy of the state prediction network output. The state prediction network generates the predicted original state at the next moment through the current original state and the predicted action. The original state at the current moment and the predicted original state at the next moment are fused according to the weight through the state fusion module. The fused state makes the state transition of the unmanned vehicle more in line with the Markov property, which can improve the model training speed and make the model converge faster. The present application corrects the state of the unmanned vehicle through the state fusion correction framework to ensure that the state transition of the unmanned vehicle satisfies the Markov property, thereby improving the convergence speed of the policy network and improving the path planning ability of the unmanned vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is an application environment diagram of an autonomous path planning method for an unmanned vehicle in one embodiment of the present application;

[0046] Figure 2 A schematic diagram of a process flow of an autonomous path planning method for an unmanned vehicle provided in one embodiment of the present application;

[0047] Figure 3 Schematic diagram of the state fusion correction framework structure. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] The autonomous path planning method for an unmanned vehicle provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the state of the unmanned vehicle to be planned to the server 104. After the server 104 receives the state of the unmanned vehicle to be planned, the server 104 uses the trained path planning model for the state of the unmanned vehicle to be planned to obtain the execution action. The server 104 can feedback the obtained execution action to the terminal 102. In addition, in some embodiments, the unmanned vehicle autonomous path planning method can also be implemented separately by the server 104 or the terminal 102, such as the terminal 102 can directly process the state of the unmanned vehicle to be planned, or the server 104 can obtain the state of the unmanned vehicle to be planned from the data storage system and process the state of the unmanned vehicle to be planned.

[0051] In an exemplary embodiment, Figure 2 As shown, a method for autonomous path planning of an unmanned vehicle is provided. The method is executed by a computer device, and specifically can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following S201 to S205. Among them:

[0052] S201, construct a state fusion correction framework; Figure 3 As shown, the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to calculate the state s at time t. t Output predicted action a′ t The state prediction network is used to predict the state s at time t t and the predicted action a′ t Output predicted state s′ t+1 The state fusion module is used to transform the state s at time t t and the predicted state s′ t+1 Fusion, get the fusion state S t ; S t =αs t +(1-α)s′ t+1 ; In the formula, α is a hyperparameter, which is used to indicate the importance of the original state and the predicted state in the fusion state; specifically α = 0.5;

[0053] The input of the action prediction network is the original state information, and the input of the policy network is the fusion state. Therefore, the structures of the two networks cannot be exactly the same, that is, the middle layer of the action prediction network has n more layers than the middle layer of the policy network. In order to make the action prediction retain the calculation characteristics of the policy network to ensure that the predicted action conforms to the real distribution, the remaining neural network layers of the action prediction network share the parameters of the policy network, n = 1, 2, 3...;

[0054] In an exemplary embodiment, the action prediction network has two more intermediate layers than the policy network, and the remaining neural network layers share the policy network parameters. The action prediction network input channels are set to 20, 8 fully connected intermediate layers, the activation function uses the ReLU function, and the output channel is 2. The state prediction network input channels are 22, 6 fully connected intermediate layers, the activation function uses the ReLU function, and the output channel is 20.

[0055] The action space of the PPO algorithm is expressed as: a t =(v t ,ω t );where v t represents the linear speed of the unmanned vehicle, ω tRepresents the angular velocity of the unmanned vehicle. The PPO algorithm can output continuous values, so the action space does not need to be discretized.

[0056] In order to ensure the completeness of information and to simplify the structure, it is convenient for the prediction of the state fusion correction framework. The state includes: the distance between the unmanned vehicle and the target, the position of the target in the unmanned vehicle, and the radar point cloud information; that is, s t =(d t , ρ t , Randar t ), where d t represents the distance between the unmanned vehicle and the target at time t, ρ t Indicates the position of the target at time t in the direction of the unmanned vehicle, Radar t Represents the radar point cloud information at time t, and obtains obstacle information based on the point cloud data, ρ t It is expressed as:

[0057]

[0058] S202, constructing a policy network based on a deep reinforcement learning algorithm; the policy network is used to output an action to be executed according to the fusion state;

[0059] S203, determining a path planning model according to the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output;

[0060] S204, determining the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle, and the radar point cloud information;

[0061] S205, according to the state of the unmanned vehicle to be planned, use the trained path planning model to obtain the execution action.

[0062] Implement S201-S205, build a strategy network through the PPO algorithm, and be able to output continuous action values, ensure the continuity of state mapping actions, and improve the path planning effect of the unmanned vehicle; construct a state fusion correction framework, design an action prediction network, and predict actions through the original state at the current moment, providing a basis for the state prediction network, ensuring the accuracy of the state prediction network output. Design a state prediction network, generate the predicted original state at the next moment through the current original state and predicted action, and fuse the original state at the current moment with the predicted original state at the next moment according to the weight through the state fusion module. The fusion state makes the state transfer of the unmanned vehicle more in line with Markov properties, which can improve the model training speed and make the model converge faster.

[0063] In an exemplary embodiment, the target position and the unmanned vehicle position are randomly initialized for each training. If the target point position or the unmanned vehicle initial position coincides with a static obstacle or is located on the motion trajectory of a dynamic obstacle, it is reinitialized.

[0064] The target position G is expressed as:

[0065] G=(x g ,y g );

[0066] In the formula, x g Indicates the horizontal coordinate of the target position, y g The vertical coordinate representing the target position;

[0067] Unmanned vehicle position C t It is expressed as:

[0068]

[0069] In the formula, The horizontal coordinate representing the position of the unmanned vehicle, The vertical coordinate representing the position of the unmanned vehicle;

[0070] The reward function r of the policy network t It is expressed as:

[0071]

[0072] Among them, the “other cases” in the reward function are the states where the unmanned vehicle moves to find the target, r d represents the distance reward, r d =tanh(d t-1 -d t ), the tanh function is used for normalization. The closer the position of the unmanned vehicle is to the target position, the greater the distance reward the unmanned vehicle receives. t represents the distance between the unmanned vehicle and the target at time t, d t-1 represents the distance between the unmanned vehicle and the target at time t-1, x g Indicates the horizontal coordinate of the target position, y g The ordinate of the target position. The horizontal coordinate representing the position of the unmanned vehicle, The ordinate of the unmanned vehicle’s position, r v Indicates line speed bonus, v t Indicates the linear velocity, through v max and v min Normalized, the greater the speed of the unmanned vehicle, the greater the speed reward. The speed reward can ensure that the unmanned vehicle maintains a high speed to approach the target position, vmin Indicates the minimum linear velocity, v max Indicates the maximum linear velocity, r ω represents the angular velocity reward, By Omega max Normalized, the greater the angular velocity of the unmanned vehicle, the smaller the reward. The angular velocity reward can limit the unmanned vehicle from turning, ensuring that the unmanned vehicle is heading in the target direction and reducing unnecessary turning actions. t represents the angular velocity, ω max represents the maximum angular velocity, r c represents the state error reward, r c = -tanh(|s t+1 -s′ t+1 |), normalized by tanh function, the smaller the gap between the predicted state and the actual state, the greater the reward, s t+1 represents the state at time t+1, s′ t+1 Represents the predicted state at time t+1.

[0073] The reward function in this application is composed of distance reward, speed reward, angular velocity reward, and state error reward. It can limit the training of the model from multiple aspects and improve the effect of model training. At the same time, the reward functions of each part are normalized to prevent the reward function of a certain part from having too much influence on the model due to scale problems, thereby improving the stability of the model.

[0074] The objective function J(θ) of the policy network is expressed as:

[0075]

[0076] Among them, p θ (a t |s t ) represents the action output probability of the policy network, represents the action output probability of the policy network before updating, θ represents the parameters of the policy network, τ represents a batch of samples, b represents the expectation of reward, γ is a hyperparameter, and k represents the number of times the model is updated for each batch of samples. represents the intermediate parameter, R(τ)=r1+γr2+γ 2 r3+...+γ n-1 r n , clip() represents the program function, ε represents the hyperparameter, R(τ) represents the total reward value obtained by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle, r1, r2, r3 and r n It represents the reward value obtained by the unmanned vehicle when performing the corresponding action, and n represents the number of actions performed by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle.

[0077] In an exemplary embodiment, γ=0.1, k=6, the policy network has 20 input channels, 6 fully connected intermediate layers, the activation function uses the ReLU function, and the output channel is 2; the policy network is trained by the loss of J(θ)

[0078] The training process of the policy network is as follows:

[0079] S501: Initialize the network parameters in the strategy network and state fusion correction framework;

[0080] S502: The distance d between the unmanned vehicle and the target t , the position of the dynamic target point in the unmanned vehicle ρ t , radar point cloud information at time t t Get the original state s of the unmanned vehicle at time t t ;

[0081] S503: The original state s at time t t Input into the state fusion correction framework, and obtain the fusion state S through the action prediction network, state prediction network, and state fusion module t ;

[0082] S504: merge the state S t Input into the policy network, the policy network outputs action a t , the unmanned vehicle performs action a t , returns the reward value r t ;

[0083] S505: Obtain the fusion state S at time t+1 through S502 to S503 t+1 ;

[0084] S506: (S t , a t , r t , S t+1 ) is stored in the experience replay pool;

[0085] S507: Repeat S502 to S506 to obtain a certain number of samples;

[0086] S508: Update network parameters after the number of samples reaches the set threshold;

[0087] S509: Repeat S502 to S508 until the average reward value of the unmanned vehicle reaches the set threshold and the training ends.

[0088] The set threshold may be a success threshold, which is a hyperparameter. In a specific embodiment, the success threshold is set to 0.2 meters, that is, when the distance between the unmanned vehicle and the target is less than or equal to 0.2 meters, the unmanned vehicle reaches the target position.

[0089] The training process of the path planning model also includes:

[0090] Create a ground training and verification simulation environment; use Gazebo under the Linux system to create a ground training and verification simulation environment;

[0091] Based on the same inventive concept, the embodiment of the present application also provides an unmanned vehicle autonomous path planning device for implementing the unmanned vehicle autonomous path planning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more unmanned vehicle autonomous path planning device embodiments provided below can refer to the limitations of the unmanned vehicle autonomous path planning method above, and will not be repeated here.

[0092] In an exemplary embodiment, an unmanned vehicle autonomous path planning device is provided, comprising:

[0093] The state fusion correction framework construction unit is used to construct the state fusion correction framework; the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to output a predicted action according to the state at time t; the state prediction network is used to output a predicted state according to the state at time t and the predicted action; the state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state; the state includes: the distance between the unmanned vehicle and the target, the position of the target position in the unmanned vehicle and the radar point cloud information;

[0094] A policy network construction unit, used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to perform actions according to the fusion state output;

[0095] A path planning model determination unit, used to determine a path planning model according to the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output;

[0096] The state determination unit of the unmanned vehicle is used to determine the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle and the radar point cloud information;

[0097] The path planning unit is used to obtain the execution action according to the state of the unmanned vehicle to be planned using the trained path planning model.

[0098] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for autonomous path planning of an unmanned vehicle is implemented.

[0099] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0100] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0103] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0104] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An autonomous path planning method for an unmanned vehicle, characterized in that: The unmanned vehicle autonomous path planning method comprises: Construct a state fusion correction framework; the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to output a predicted action according to the state at time t; the state prediction network is used to output a predicted state according to the state at time t and the predicted action; the state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state; the state includes: the distance between the unmanned vehicle and the target, the position of the target position in the unmanned vehicle and the radar point cloud information; Based on the deep reinforcement learning algorithm, a policy network is constructed; the policy network is used to perform actions according to the fusion state output; Determine a path planning model based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output; Determine the status of the unmanned vehicle to be planned based on the target location, the unmanned vehicle location, and the radar point cloud information; According to the state of the unmanned vehicle to be planned, the trained path planning model is used to obtain the execution action.

2. The autonomous path planning method for an unmanned vehicle according to claim 1, characterized in that: The reward function r of the policy network t It is expressed as: Among them, r d represents the distance reward, r d =tanh(d t-1 -d t ), the tanh function is used for normalization, d t represents the distance between the unmanned vehicle and the target at time t, d t-1 represents the distance between the unmanned vehicle and the target at time t-1, x g Indicates the horizontal coordinate of the target position, y g The ordinate of the target position. The horizontal coordinate representing the position of the unmanned vehicle, The ordinate of the unmanned vehicle position, r v Indicates line speed bonus, v t represents the linear velocity, v min Indicates the minimum linear velocity, v max Indicates the maximum linear velocity, r ω represents the angular velocity reward, ω t represents the angular velocity, ω max represents the maximum angular velocity, r c represents the state error reward, r c = -tanh(|s t+1 -s′ t+1 |),s t+1 represents the state at time t+1, s′ t+1 Represents the predicted state at time t+1.

3. The autonomous path planning method for an unmanned vehicle according to claim 1, characterized in that: The objective function J(θ) of the policy network is expressed as: Among them, p θ (a t |s t ) represents the action output probability of the policy network, represents the action output probability of the policy network before updating, θ represents the parameters of the policy network, τ represents a batch of samples, b represents the expectation of reward, γ is a hyperparameter, and k represents the number of times the model is updated for each batch of samples. represents the intermediate parameter, R(τ)=r1+γr2+γ 2 r3+…+γ n-1 r n , clip() represents the program function, ε represents the hyperparameter, R(τ) represents the total reward value obtained by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle, r1, r2, r3 and r n It represents the reward value obtained by the unmanned vehicle when performing the corresponding action, and n represents the number of actions performed by the unmanned vehicle from starting to finding the target or from starting to hitting the obstacle.

4. The autonomous path planning method for an unmanned vehicle according to claim 1, characterized in that: The training process of the path planning model is: Create ground training and validation simulation environments; Initialize the network parameters in the policy network and state fusion correction framework; Get the original state of the unmanned vehicle at time t; The original state at time t is input into the state fusion correction framework, and the fusion state is obtained through the action prediction network, state prediction network and state fusion module; The fusion state is input into the policy network, the policy network outputs the action to be executed, the unmanned vehicle performs the corresponding action and returns the reward value; Obtain the fusion state at time t+1, and store the fusion state at time t, the executed action, the reward value, and the fusion state at time t+1 in the experience replay pool; until a set number of samples are obtained, update the parameters of the path planning model; The training is completed until the average reward value of the driverless car reaches the set threshold.

5. The autonomous path planning method for an unmanned vehicle according to claim 4, characterized in that: Create a ground-based training and validation simulation environment, including: Use Gazebo under Linux system to create a ground training and verification simulation environment.

6. The autonomous path planning method for an unmanned vehicle according to claim 1, characterized in that: Determining the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle, and the radar point cloud information specifically includes: Determine the distance between the unmanned vehicle and the target based on the target position and the unmanned vehicle position; Using the formula Determine the position of the target position at time t in the direction of the unmanned vehicle ρ t , x g Indicates the horizontal coordinate of the target position, y g The ordinate of the target position. The horizontal coordinate representing the position of the unmanned vehicle, The vertical coordinate representing the position of the unmanned vehicle.

7. An autonomous path planning device for an unmanned vehicle, characterized in that: The unmanned vehicle autonomous path planning device comprises: The state fusion correction framework construction unit is used to construct the state fusion correction framework; the state fusion correction framework includes: an action prediction network, a state prediction network and a state fusion module; the action prediction network is used to output a predicted action according to the state at time t; the state prediction network is used to output a predicted state according to the state at time t and the predicted action; the state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state; the state includes: the distance between the unmanned vehicle and the target, the position of the target position in the unmanned vehicle and the radar point cloud information; A policy network construction unit, used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to perform actions according to the fusion state output; A path planning model determination unit, used to determine a path planning model according to the state fusion correction framework and the policy network; the path planning model takes the current state as input and the executed action as output; The state determination unit of the unmanned vehicle is used to determine the state of the unmanned vehicle to be planned according to the target position, the position of the unmanned vehicle and the radar point cloud information; The path planning unit is used to obtain the execution action according to the state of the unmanned vehicle to be planned using the trained path planning model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous path planning method for an unmanned vehicle according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the unmanned vehicle autonomous path planning method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the unmanned vehicle autonomous path planning method described in any one of claims 1 to 6 is implemented.

Images