A method, device, medium, and product for autonomous path planning of unmanned vehicles

Abstract

This application discloses an autonomous path planning method, device, medium, and product for unmanned vehicles, relating to the field of autonomous driving technology. The method includes constructing a state fusion correction framework; the state fusion correction framework includes an action prediction network, a state prediction network, and a state fusion module; constructing a policy network based on a deep reinforcement learning algorithm; the policy network is used to output actions to be executed based on the fused state; determining a path planning model based on the state fusion correction framework and the policy network; determining the state of the unmanned vehicle to be planned based on the target location, the unmanned vehicle's location, and radar point cloud information; and obtaining the actions to be executed based on the state of the unmanned vehicle to be planned using the trained path planning model. This application can improve the path planning capability of unmanned vehicles.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous path planning method, device, medium and product for unmanned vehicles. Background Technology

[0002] Autonomous vehicles (RVs) represent a significant application of outdoor mobile robots in the transportation sector. As one of the most widely used unmanned systems, RVs can accomplish most ground-based tasks, including target search, disaster relief, and logistics transportation. In the vast majority of these tasks, RVs require path planning capabilities; therefore, path planning is a fundamental and crucial function for RVs to complete complex tasks.

[0003] The A* algorithm is a widely used heuristic algorithm in path planning, and many researchers have improved upon it based on the kinematic characteristics of autonomous vehicles. However, the actual task environments of autonomous vehicles are complex and varied, making it difficult for these traditional algorithms to accurately describe their dynamic models. Furthermore, as the complexity of the environment increases, these algorithms require more computational resources.

[0004] Therefore, path planning for autonomous vehicles faces challenges such as difficulties in kinematic and dynamic modeling and dynamic changes in controller parameters. The development of reinforcement learning technology has opened up new avenues for solving this problem. Reinforcement learning does not require kinematic and dynamic modeling of the autonomous vehicle; it allows the agent to continuously interact with the environment and evaluate the agent's actions using a reward function, gradually learning a complete control policy. However, in the reinforcement learning framework, the Markov property of the agent's state transitions is a prerequisite for ensuring convergence during training. In real-world control tasks, however, many factors can cause the agent's state transitions to not satisfy the Markov property. On one hand, policy generation and action execution require computation time; more complex policies or environments often require more computational resources and time to generate actions. On the other hand, the autonomous vehicle also needs time to respond to action commands, leading to state transitions that do not conform to the Markov property, further resulting in slow network convergence and delayed path planning.

[0005] Therefore, in order to improve the path planning capabilities of autonomous vehicles, there is an urgent need to provide an autonomous path planning method or device for autonomous vehicles. Summary of the Invention

[0006] The purpose of this application is to provide an autonomous path planning method, device, medium, and product for unmanned vehicles, which can improve the path planning capabilities of unmanned vehicles.

[0007] To achieve the above objectives, this application provides the following solution:

[0008] Firstly, this application provides an autonomous path planning method for unmanned vehicles, the autonomous path planning method for unmanned vehicles comprising:

[0009] A state fusion correction framework is constructed, comprising: an action prediction network, a state prediction network, and a state fusion module. The action prediction network outputs a predicted action based on the state at time t. The state prediction network outputs a predicted state based on the state at time t and the predicted action. The state fusion module fuses the state at time t with the predicted state to obtain a fused state. The state includes: the distance between the unmanned vehicle and the target, the target's position relative to the unmanned vehicle, and radar point cloud information.

[0010] A policy network is constructed based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state.

[0011] The path planning model is determined based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the action to be executed as output.

[0012] The status of the unmanned vehicle to be planned is determined based on the target location, the unmanned vehicle's location, and radar point cloud information.

[0013] Based on the state of the autonomous vehicle to be planned, a trained path planning model is used to obtain the actions to be executed.

[0014] Optionally, the reward function r of the policy network t Represented as:

[0015]

[0016] Where, r d Represents the distance reward, r d =tand(d t-1 -d t The tanh function is used for normalization, d t Let d represent the distance between the autonomous vehicle and the target at time t. t-1 This represents the distance between the unmanned vehicle and the target at time t-1. x g The x-coordinate of the target location, y g The vertical coordinate represents the target's location. The x-coordinate represents the location of the autonomous vehicle. The ordinate r represents the position of the autonomous vehicle. v Indicates linear velocity bonus. v t V represents linear velocity. min v represents the minimum linear velocity. maxThe value of the linear velocity, r ω Indicates angular velocity bonus. ω t ω represents angular velocity. max The value of angular velocity is represented by r. c The state error reward, r c =-tanh(|s t+1 -s′ t+1 |), s t+1 Let s' represent the state at time t+1. t+1 This represents the predicted state at time t+1.

[0017] Optionally, the objective function J(θ) of the policy network is expressed as:

[0018]

[0019] Where, p θ (a t |s t ) represents the probability of the action output of the policy network. Let θ represent the action output probability of the policy network before the update, τ represent the parameters of the policy network, b represent the expected reward, γ represent the hyperparameters, and k represent the number of times the model is updated per batch of samples. Indicates intermediate parameters. R(τ)=r1+γr2+γ 2 r3+...+γ n-1 r n `clip()` represents a program function, `ε` represents hyperparameters, `R(τ)` represents the total reward value obtained by the autonomous vehicle for each cycle from startup to finding the target or from startup to hitting an obstacle, and `r1`, `r2`, `r3`, and `r` are also mentioned. n This represents the reward value obtained by the autonomous vehicle in performing the corresponding action, and n represents the number of actions performed by the autonomous vehicle from startup to finding the target or from startup to hitting the obstacle.

[0020] Optionally, the training process of the path planning model is as follows:

[0021] Create a ground-based training and verification simulation environment;

[0022] Initialize the network parameters in the policy network and the state fusion correction framework;

[0023] Obtain the original state of the autonomous vehicle at time t;

[0024] The original state at time t is input into the state fusion correction framework, and the fused state is obtained through the action prediction network, the state prediction network, and the state fusion module.

[0025] The fused state is input into the policy network, the policy network outputs the action to be executed, the autonomous vehicle executes the corresponding action, and returns the reward value;

[0026] Obtain the fusion state at time t+1, and store the fusion state at time t, the executed action, the reward value, and the fusion state at time t+1 in the experience replay pool; until a set number of samples are obtained, update the parameters of the path planning model.

[0027] Training continues until the average reward value of the autonomous vehicle reaches a set threshold, at which point the training is complete.

[0028] Optionally, a ground training and validation simulation environment is created, including:

[0029] We used Gazebo under the Linux system to create a ground training and verification simulation environment.

[0030] Optionally, determining the state of the unmanned vehicle to be planned based on the target location, the unmanned vehicle's location, and radar point cloud information specifically includes:

[0031] The distance between the autonomous vehicle and the target is determined based on the target location and the autonomous vehicle's location;

[0032] Using formula Determine the target's position ρ at time t relative to the autonomous vehicle's orientation. t x g The x-coordinate of the target location, y g The vertical coordinate represents the target's location. The x-coordinate represents the location of the autonomous vehicle. The vertical coordinate represents the location of the driverless vehicle.

[0033] Secondly, this application provides an autonomous path planning device for unmanned vehicles, the autonomous path planning device comprising:

[0034] A state fusion correction framework construction unit is used to construct a state fusion correction framework. The state fusion correction framework includes: an action prediction network, a state prediction network, and a state fusion module. The action prediction network is used to output a predicted action based on the state at time t. The state prediction network is used to output a predicted state based on the state at time t and the predicted action. The state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state. The state includes: the distance between the unmanned vehicle and the target, the target's position relative to the unmanned vehicle, and radar point cloud information.

[0035] A policy network construction unit is used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state.

[0036] The path planning model determination unit is used to determine the path planning model based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the action to be executed as output.

[0037] The autonomous vehicle state determination unit is used to determine the state of the autonomous vehicle to be planned based on the target position, the autonomous vehicle position, and radar point cloud information.

[0038] The path planning unit is used to determine the actions to be executed based on the state of the autonomous vehicle to be planned, using a trained path planning model.

[0039] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous path planning method for unmanned vehicles.

[0040] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the autonomous path planning method for unmanned vehicles.

[0041] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the autonomous path planning method for unmanned vehicles.

[0042] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0043] This application provides an autonomous path planning method, device, medium, and product for unmanned vehicles. Based on a deep reinforcement learning (Proximal Policy Optimization, PPO) algorithm, a policy network is constructed, which can output continuous action values, ensuring the continuity of state-to-action mapping and improving the path planning performance of the unmanned vehicle. In the constructed state fusion correction framework, the action prediction network predicts actions based on the current state, providing a basis for the state prediction network and ensuring the accuracy of its output. The state prediction network generates the predicted original state for the next time step based on the current original state and the predicted action. The state fusion module fuses the current original state and the predicted original state for the next time step according to weights. This fused state makes the state transition of the unmanned vehicle more conform to the Markov property, improving model training speed and enabling faster model convergence. This application corrects the state of the unmanned vehicle through the state fusion correction framework, ensuring that the state transition of the unmanned vehicle satisfies the Markov property, thereby improving the convergence speed of the policy network and enhancing the path planning capability of the unmanned vehicle. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an application environment diagram of an autonomous path planning method for unmanned vehicles according to an embodiment of this application;

[0046] Figure 2 A flowchart illustrating an autonomous path planning method for unmanned vehicles provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of the state fusion correction framework structure. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] The autonomous path planning method for unmanned vehicles provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the status of the unmanned vehicle to be planned to server 104. After receiving the status of the unmanned vehicle to be planned, server 104 uses a trained path planning model to obtain the action to be executed. Server 104 can then feed back the executed action to terminal 102. Furthermore, in some embodiments, the autonomous path planning method for the unmanned vehicle can also be implemented independently by server 104 or terminal 102. For example, terminal 102 can directly process the status of the unmanned vehicle to be planned, or server 104 can obtain the status of the unmanned vehicle to be planned from the data storage system and process it accordingly.

[0051] In one exemplary embodiment, such as Figure 2 As shown, an autonomous path planning method for unmanned vehicles is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S205. Among them:

[0052] S201, Construct a state fusion correction framework; such as Figure 3 As shown, the state fusion correction framework includes: an action prediction network, a state prediction network, and a state fusion module; the action prediction network is used to determine the state s at time t. t Output the predicted action a′ t The state prediction network is used to predict the state s at time t. t and the predicted action a′ t Output predicted state s′ t+1 The state fusion module is used to convert the state s at time t. t With the predicted state s′ t+1 The mixture is fused to obtain a fused state S. t S t =αs t +(1-α)s′ t+1 In the formula, α is a hyperparameter used to represent the importance of the original state and the predicted state in the fused state; specifically, α = 0.5.

[0053] The input to the action prediction network is the original state information, and the input to the policy network is the fused state. Therefore, the structures of the two networks cannot be exactly the same. That is, the intermediate layers of the action prediction network are n more than the intermediate layers of the policy network. In order to preserve the computational characteristics of the policy network to ensure that the predicted action conforms to the true distribution, the remaining neural network layers of the action prediction network share the parameters of the policy network, n = 1, 2, 3...

[0054] In one exemplary embodiment, the action prediction network has two more intermediate layers than the policy network, and the remaining neural network layers share the parameters of the policy network. The action prediction network has 20 input channels, 8 fully connected intermediate layers, uses the ReLU activation function, and has 2 output channels. The state prediction network has 22 input channels, 6 fully connected intermediate layers, uses the ReLU activation function, and has 20 output channels.

[0055] The action space of the PPO algorithm is represented as: a t =(v t ω t In the formula, v t ω represents the linear velocity of the autonomous vehicle. tThis represents the angular velocity of the autonomous vehicle. The PPO algorithm can output continuous values, so the action space does not need to be discretized.

[0056] To ensure information completeness and maintain a simple structure for easy prediction by the state fusion correction framework, the states include: the distance between the autonomous vehicle and the target, the target's position relative to the autonomous vehicle, and radar point cloud information; i.e., s t =(d t , ρ t Randar t In the formula, d t ρ represents the distance between the autonomous vehicle and the target at time t. t This indicates the target's location at time t relative to the autonomous vehicle's position. (Radar) t ρ represents the radar point cloud information at time t, and obstacle information is obtained from the point cloud data. t Represented as:

[0057]

[0058] S202, Construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state;

[0059] S203, determine the path planning model based on the state fusion correction framework and policy network; the path planning model takes the current state as input and the action to be executed as output;

[0060] S204: Determine the status of the unmanned vehicle to be planned based on the target location, the unmanned vehicle's location, and radar point cloud information;

[0061] S205: Based on the state of the unmanned vehicle to be planned, a trained path planning model is used to obtain the actions to be executed.

[0062] Implementing S201-S205, a policy network is built using the PPO algorithm, which outputs continuous action values, ensuring the continuity of state-to-action mapping and improving the path planning performance of the autonomous vehicle. A state fusion correction framework is constructed, and an action prediction network is designed. The network predicts actions based on the current state, providing a basis for the state prediction network's output and ensuring accuracy. The state prediction network generates the predicted initial state for the next time step from the current initial state and the predicted action. A state fusion module fuses the current initial state and the predicted initial state for the next time step according to weights. This fused state makes the autonomous vehicle's state transitions more Markov-like, improving model training speed and enabling faster model convergence.

[0063] In an exemplary embodiment, the target position and the autonomous vehicle position are randomly initialized during each training session, and are re-initialized if the target position or the initial position of the autonomous vehicle coincides with a static obstacle or is located on the trajectory of a dynamic obstacle.

[0064] The target position G is represented as:

[0065] G=(x g ,y g );

[0066] In the formula, x g The x-coordinate of the target location, y g The vertical coordinate representing the target's location;

[0067] Unmanned vehicle location C t Represented as:

[0068]

[0069] In the formula, The x-coordinate represents the location of the autonomous vehicle. The vertical coordinate represents the position of the autonomous vehicle;

[0070] The reward function r of the policy network t Represented as:

[0071]

[0072] In the reward function, "other cases" refers to the state in which the autonomous vehicle moves to find the target, r d Represents the distance reward, r d =tanh(d t-1 -d t The tanh function is used for normalization. The closer the autonomous vehicle's position is to the target position, the greater the distance reward it receives. t Let d represent the distance between the autonomous vehicle and the target at time t. t-1 This represents the distance between the unmanned vehicle and the target at time t-1. x g The x-coordinate of the target location, y g The vertical coordinate represents the target's location. The x-coordinate represents the location of the autonomous vehicle. The ordinate r represents the position of the autonomous vehicle. v Indicates linear velocity bonus. v t Represents linear velocity, expressed by v max and v min Normalization is applied; the higher the speed of the autonomous vehicle, the greater the speed bonus. This speed bonus ensures the autonomous vehicle maintains a high speed to approach the target location.min v represents the minimum linear velocity. max The value of the linear velocity, r ω Indicates angular velocity bonus. Through ω max Normalization is performed; the higher the angular velocity of the autonomous vehicle, the smaller the reward. Angular velocity rewards can limit the autonomous vehicle's turning, ensuring that the vehicle's orientation is towards the target direction and reducing unnecessary turning maneuvers. ω t ω represents angular velocity. max The value of angular velocity is represented by r. c The state error reward, r c =-tanh(|s t+1 -s′ t+1 |), normalized using the tanh function, the smaller the difference between the predicted state and the true state, the greater the reward. t+1 Let s' represent the state at time t+1. t+1 This represents the predicted state at time t+1.

[0073] The reward function in this application consists of distance reward, velocity reward, angular velocity reward, and state error reward, which can restrict the training of the model from multiple aspects and improve the training effect. At the same time, the reward function of each part is normalized to prevent the reward function of a certain part from having an excessive impact on the model due to scale issues, thereby improving the stability of the model.

[0074] The objective function J(θ) of the policy network is expressed as:

[0075]

[0076] Where, p θ (a t |s t ) represents the probability of the action output of the policy network. Let θ represent the action output probability of the policy network before the update, τ represent the parameters of the policy network, b represent the expected reward, γ represent the hyperparameters, and k represent the number of times the model is updated per batch of samples. Indicates intermediate parameters. R(τ)=r1+γr2+γ 2 r3+...+γ n-1 r n `clip()` represents a program function, `ε` represents hyperparameters, `R(τ)` represents the total reward value obtained by the autonomous vehicle for each cycle from startup to finding the target or from startup to hitting an obstacle, and `r1`, `r2`, `r3`, and `r` are also mentioned. n This represents the reward value obtained by the autonomous vehicle in performing the corresponding action, and n represents the number of actions performed by the autonomous vehicle from startup to finding the target or from startup to hitting the obstacle.

[0077] In an exemplary embodiment, γ = 0.1, k = 6, the policy network has 20 input channels, 6 fully connected intermediate layers, uses the ReLU activation function, and has 2 output channels; the policy network is trained using the loss function J(θ).

[0078] The training process of the policy network is as follows:

[0079] S501: Initialize the network parameters in the policy network and state fusion correction framework;

[0080] S502: The distance d between the autonomous vehicle and the target t The location ρ of the dynamic target point on the autonomous vehicle t Radar point cloud information at time t t Obtain the initial state s of the autonomous vehicle at time t. t ;

[0081] S503: The original state s at time t t The input is fed into the state fusion correction framework, and the fused state S is obtained through the action prediction network, state prediction network, and state fusion module. t ;

[0082] S504: Merge state S t The input is fed into the policy network, and the policy network outputs action a. t The driverless car performs action a t Returns reward value r t ;

[0083] S505: Obtain the fusion state S at time t+1 through S502 to S503. t+1 ;

[0084] S506: (S t , a t r t S t+1 Stored in the experience replay pool;

[0085] S507: Repeat S502 to S506 to obtain a certain number of samples;

[0086] S508: Update network parameters after the number of samples reaches a set threshold;

[0087] S509: Repeat S502 to S508 until the average reward value of the autonomous vehicle reaches the set threshold, and the training ends.

[0088] The threshold can be set as a success threshold, which is a hyperparameter. In a specific embodiment, the success threshold is set to 0.2 meters, that is, when the distance between the unmanned vehicle and the target is less than or equal to 0.2 meters, the unmanned vehicle reaches the target position.

[0089] The training process of the path planning model also includes:

[0090] Create a ground training and verification simulation environment; specifically, use Gazebo under the Linux system to create the ground training and verification simulation environment.

[0091] Based on the same inventive concept, this application also provides an autonomous path planning device for implementing the autonomous path planning method for unmanned vehicles as described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the autonomous path planning device provided below can be found in the limitations of the autonomous path planning method described above, and will not be repeated here.

[0092] In one exemplary embodiment, an autonomous path planning device for unmanned vehicles is provided, comprising:

[0093] A state fusion correction framework construction unit is used to construct a state fusion correction framework. The state fusion correction framework includes: an action prediction network, a state prediction network, and a state fusion module. The action prediction network is used to output a predicted action based on the state at time t. The state prediction network is used to output a predicted state based on the state at time t and the predicted action. The state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state. The state includes: the distance between the unmanned vehicle and the target, the target's position relative to the unmanned vehicle, and radar point cloud information.

[0094] A policy network construction unit is used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state.

[0095] The path planning model determination unit is used to determine the path planning model based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the action to be executed as output.

[0096] The autonomous vehicle state determination unit is used to determine the state of the autonomous vehicle to be planned based on the target position, the autonomous vehicle position, and radar point cloud information.

[0097] The path planning unit is used to determine the actions to be executed based on the state of the autonomous vehicle to be planned, using a trained path planning model.

[0098] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an autonomous path planning method for unmanned vehicles.

[0099] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0100] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0103] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An autonomous path planning method for unmanned vehicles, characterized in that, The autonomous path planning method for unmanned vehicles includes: A state fusion correction framework is constructed, comprising: an action prediction network, a state prediction network, and a state fusion module. The action prediction network outputs a predicted action based on the state at time t. The state prediction network outputs a predicted state based on the state at time t and the predicted action. The state fusion module fuses the state at time t with the predicted state to obtain a fused state. The state includes: the distance between the unmanned vehicle and the target, the target's position relative to the unmanned vehicle, and radar point cloud information. A policy network is constructed based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state. The path planning model is determined based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the action to be executed as output. The status of the unmanned vehicle to be planned is determined based on the target location, the unmanned vehicle's location, and radar point cloud information. Based on the state of the autonomous vehicle to be planned, a trained path planning model is used to obtain the actions to be executed. The reward function r of the policy network t Represented as: Where, r d Represents the distance reward, r d =tanh(d t-1 -d t The tanh function is used for normalization, d t Let d represent the distance between the autonomous vehicle and the target at time t. t-1 This represents the distance between the unmanned vehicle and the target at time t-1. x g The x-coordinate of the target location, y g The vertical coordinate represents the target's location. The x-coordinate represents the location of the autonomous vehicle. The ordinate r represents the position of the autonomous vehicle. v Indicates linear velocity bonus. v t V represents linear velocity. min v represents the minimum linear velocity. max The value of the linear velocity, r ω Indicates angular velocity bonus. ω t ω represents angular velocity. max The value of angular velocity is represented by r. c The state error reward, r c =-tanh(|s t+1 -s′ t+1 |), s t+1 Let s' represent the state at time t+1. t+1 This represents the predicted state at time t+1; The objective function J(θ) of the policy network is expressed as: Where, p θ (a t |s t ) represents the probability of the action output of the policy network. Let θ represent the action output probability of the policy network before the update, τ represent the parameters of the policy network, b represent the expected reward, γ represent the hyperparameters, and k represent the number of times the model is updated per batch of samples. Indicates intermediate parameters. R(τ)=r1+γr2+γ 2 r3+…+γ n-1 r n `clip()` represents a program function, `ε` represents hyperparameters, `R(τ)` represents the total reward value obtained by the autonomous vehicle for each cycle from startup to finding the target or from startup to hitting an obstacle, and `r1`, `r2`, `r3`, and `r` are also mentioned. n This represents the reward value obtained by the autonomous vehicle in performing the corresponding action, and n represents the number of actions performed by the autonomous vehicle from startup to finding the target or from startup to hitting the obstacle.

2. The autonomous path planning method for unmanned vehicles according to claim 1, characterized in that, The training process of the path planning model is as follows: Create a ground-based training and verification simulation environment; Initialize the network parameters in the policy network and the state fusion correction framework; Obtain the original state of the autonomous vehicle at time t; The original state at time t is input into the state fusion correction framework, and the fused state is obtained through the action prediction network, the state prediction network, and the state fusion module. The fused state is input into the policy network, the policy network outputs the action to be executed, the autonomous vehicle executes the corresponding action, and returns the reward value; Obtain the fusion state at time t+1, and store the fusion state at time t, the executed action, the reward value, and the fusion state at time t+1 in the experience replay pool; until a set number of samples are obtained, update the parameters of the path planning model. Training continues until the average reward value of the autonomous vehicle reaches a set threshold, at which point the training is complete.

3. The autonomous path planning method for unmanned vehicles according to claim 2, characterized in that, Create a ground training and validation simulation environment, including: We used Gazebo under the Linux system to create a ground training and verification simulation environment.

4. The autonomous path planning method for unmanned vehicles according to claim 1, characterized in that, The process of determining the state of the unmanned vehicle to be planned based on the target location, the unmanned vehicle's location, and radar point cloud information specifically includes: The distance between the autonomous vehicle and the target is determined based on the target location and the autonomous vehicle's location; Using formula Determine the target's position ρ at time t relative to the autonomous vehicle's orientation. t x g The x-coordinate of the target location, y g The vertical coordinate represents the target's location. The x-coordinate represents the location of the autonomous vehicle. The vertical coordinate represents the location of the driverless vehicle.

5. An autonomous path planning device for unmanned vehicles, used to implement the autonomous path planning device for unmanned vehicles as described in any one of claims 1-4, characterized in that, The autonomous path planning equipment for the unmanned vehicle includes: A state fusion correction framework construction unit is used to construct a state fusion correction framework. The state fusion correction framework includes: an action prediction network, a state prediction network, and a state fusion module. The action prediction network is used to output a predicted action based on the state at time t. The state prediction network is used to output a predicted state based on the state at time t and the predicted action. The state fusion module is used to fuse the state at time t with the predicted state to obtain a fused state. The state includes: the distance between the unmanned vehicle and the target, the target's position relative to the unmanned vehicle, and radar point cloud information. A policy network construction unit is used to construct a policy network based on a deep reinforcement learning algorithm; the policy network is used to output the action to be executed according to the fusion state. The path planning model determination unit is used to determine the path planning model based on the state fusion correction framework and the policy network; the path planning model takes the current state as input and the action to be executed as output. The autonomous vehicle state determination unit is used to determine the state of the autonomous vehicle to be planned based on the target position, the autonomous vehicle position, and radar point cloud information. The path planning unit is used to determine the actions to be executed based on the state of the autonomous vehicle to be planned, using a trained path planning model.

6. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the autonomous path planning method for unmanned vehicles as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the autonomous path planning method for unmanned vehicles as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the autonomous path planning method for unmanned vehicles as described in any one of claims 1-4.

Images