Unmanned vehicle path planning method, system and device based on deep reinforcement learning

Abstract

This invention discloses a method, system, and device for autonomous vehicle path planning based on deep reinforcement learning. The invention includes: 1. acquiring regional road environment data; 2. preprocessing the regional road environment data, including road node coordinate transformation and merging of adjacent road nodes; 3. designing a composite reward function considering the actual needs of shortest path planning; 4. constructing a path planning model based on the composite reward function using a deep reinforcement learning algorithm; and training the path planning model using the preprocessed regional road environment data to obtain the trained path planning model. This invention, based on topological road environment data obtained from an open-source platform, verifies the feasibility of applying deep reinforcement learning to real-world road environments, achieving global path planning for vehicles within a road network. By focusing on improving path planning efficiency, it lays the foundation for promoting the transformation of autonomous driving from laboratory research to the market.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and relates to a method, system and device for unmanned vehicle path planning based on deep reinforcement learning. Background Technology

[0002] With advancements in computer technology, sensor accuracy, and data processing capabilities, autonomous driving technology has achieved significant breakthroughs in environmental perception and decision-making. Path planning, as a core decision-making module in autonomous driving, directly impacts vehicle navigation efficiency and commuting efficiency within the traffic network.

[0003] Traditional deep reinforcement learning methods demonstrate superior adaptability to rule-based algorithms in path planning by mapping environmental features to high-dimensional features. However, in complex road networks, traditional deep learning algorithms still face the challenges of sparse reward dilemma and insufficient representation in high-dimensional state spaces. Long-term decision-making in road networks leads to sparse reward signals and low policy convergence efficiency. Furthermore, traditional network structures have limited ability to encode road topology, affecting path optimality.

[0004] This invention focuses on improving path planning efficiency and proposes a path planning method for unmanned vehicles based on deep reinforcement learning, which can be used to realize global path planning for vehicles in road networks. Summary of the Invention

[0005] This invention addresses the problems of sparse reward dilemma and insufficient high-dimensional state space representation in traditional deep reinforcement learning path planning algorithms under road networks. It proposes a path planning method for autonomous vehicles based on deep reinforcement learning, aiming to improve path planning efficiency.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step 1: Obtain regional road environment data;

[0008] Step 2: Perform preprocessing operations on the regional road environment data, including road node coordinate transformation and merging of adjacent road nodes;

[0009] Step 3: Consider the actual needs of shortest path planning and design a composite reward function;

[0010] Step 4: Based on the composite reward function, a path planning model is constructed using a deep reinforcement learning algorithm; the path planning model is trained using preprocessed regional road environment data to obtain the trained path planning model.

[0011] The composite reward function design in step 3 consists of three parts: path efficiency reward, exploration incentive mechanism, and cycle inhibition mechanism.

[0012] Path efficiency reward: based on the path length obtained during training in the current round. Compared with the historical best path length The ratio is generated when the current path length is detected. Less than the historical best path length At that time, the historical best path update mechanism is triggered. And through preset performance reward factors Adjust path efficiency reward .

[0013]

[0014] in, This represents the path length for training in the current round. The historical best path length for the same starting and ending points. It is a performance-based reward factor.

[0015] Explore incentive mechanisms: By judging the current time step Whether the agent accesses a new node and dynamically generates rewards .

[0016]

[0017] in, Used to determine time steps Do you want to access a new node? For the exploration gain coefficient of dynamic attenuation, set:

[0018]

[0019] in, and It is a constant value, and in the initial stage, due to the time step The gain is relatively small, so a high gain coefficient is set to encourage breadth-first search. As the training progresses, Exponential decay gradually reduces the gain coefficient, guiding the strategy to migrate towards depth-first search.

[0020] Cyclic suppression mechanism: This mechanism detects repetitive oscillations in the path using a sliding time window. When a trajectory cycle is detected within the time window, a fixed penalty coefficient is applied to generate a negative reward. .

[0021]

[0022] in, For time window, This is a function indicating the repetition of the trajectory within a time window. It is a fixed penalty coefficient used to suppress the agent's cyclical behavior, which helps the agent break out of the cycle and explore new paths.

[0023] Combining the above three rewards, we obtain the final composite reward function. .

[0024]

[0025] The specific steps of step 4 are as follows:

[0026] Step 4.1: Define the state space and action space of the deep reinforcement learning algorithm;

[0027] Step 4.2: Construct the neural network architecture for the deep reinforcement learning algorithm;

[0028] Step 4.3: Construct the mathematical framework for deep reinforcement learning algorithms based on Markov decision processes;

[0029] Step 4.4: Design a progressive training strategy to train the path planning model using preprocessed road data.

[0030] The state space S in step 4.1 is a six-dimensional feature vector:

[0031]

[0032] in, Indicates the unit time step Multidimensional information within, and They are The normalized coordinates of the current node and the normalized coordinates of the target node at any given moment; express The normalized effective degree of the current node at time step 1 represents the number of walkable nodes adjacent to the current node, and is calculated as follows: ,in The maximum node degree in the road network; express The Euclidean distance between the current node and the target node at any given time.

[0033] Action space It is a discrete decision set whose size dynamically depends on the effective degree of the current node. Specifically, each optional action For each feasible path branch originating from the current node, the total number of actions equals the effective degree of the current node. ,Right now .

[0034] Step 4.2 involves building a neural network architecture, employing a multi-head attention-guided topology feature enhancement mechanism and a dual-network decoupling framework to achieve collaborative optimization of environmental perception and policy decision-making. A multi-head self-attention layer dynamically generates a node association weight matrix, weighting and fusing topology features to enhance graph structure perception capabilities. The dual-network architecture decouples the policy selection and value evaluation process, thereby achieving collaborative optimization of environmental state perception and action decision-making.

[0035] The network input is a six-dimensional feature vector, which is encoded into a high-dimensional feature vector through a fully connected layer. After being processed by a normalization layer, the input tensor is reconstructed into a 3D tensor and input to a four-head self-attention layer. A linear transformation is performed on the input tensor to generate a query vector, key vector, and value vector. Based on the query vector and key vector, the topological association strength matrix between nodes is calculated. The association weight matrix and the value vector are then weighted and summed to obtain the output feature Z. Z is then flattened and compared with the original high-dimensional feature vector. Perform residual connections to obtain features It alleviates gradient vanishing in deep networks while balancing global topology awareness with preservation of local details.

[0036] Based on this feature, independent value branches and action branches are constructed: the value branch outputs the state-value function. The value is mapped to a scalar value through a fully connected layer, representing the potential benefit of the current state in long-term path planning; the dominant branch outputs the action advantage function. Its dimension matches the action space, characterizing the relative advantage difference of each action in immediate decision-making. To avoid the absolute bias of advantage values, the final Q value is generated through decentralized aggregation rules. Q(s,a) is a state-action value function, representing the long-term expected return of choosing action a in state s. Its decentralized aggregation rule calculation process satisfies:

[0037]

[0038] in, Indicates the state Under these conditions, the average expected cumulative reward represents the intrinsic value of a state; Indicates the state Select action The relative advantage of satisfying the zero centralization constraint ; Let be the mean of the dominance function. The current state The effective number of actions below, used to maintain and Consistency is maintained to prevent gradient instability.

[0039] Simultaneously, a dual-network architecture decoupling strategy selection and value assessment process is adopted: online network Real-time response to environmental conditions, generation of action strategies, target network A stable Q-value evaluation benchmark is provided through periodic parameter synchronization. During the training phase, the target Q-value is calculated using a double-checking mechanism. :

[0040]

[0041] in, , It is a discount factor used to balance the weight of immediate rewards and long-term rewards. For online network output, This indicates that the optimal action is selected by the online network. For the output of the target network, This indicates that the Q-value of the action is evaluated by the target network. This decoupling mechanism allows the selection of the action with the maximum Q-value and the evaluation of its corresponding Q-value to be handled by the online network. and target network Perform these steps separately to suppress the systematic overestimation bias of Q value caused by the function approximation error.

[0042] Target Network parameters A delayed update mechanism is adopted, with updates at fixed intervals. Perform a hard update, online network Synchronize parameters to the target network ,Right now .

[0043] Step 4.3, based on Markov decision processes, constructs the mathematical framework for a deep reinforcement learning model. The specific steps are as follows:

[0044] Step 4.3.1: The agent observes the state in the environment. ;

[0045] Step 4.3.2: Based on the current node's effective degree Generate a subset of executable actions ;

[0046] Step 4.3.3: Based on the strategy generated by the deep neural network, the agent selects a subset of actions according to... A greedy strategy selects actions based on probability. Explore randomly, otherwise maximize. The action. After the action is executed, the environment updates its state value. ;

[0047]

[0048] Step 4.3.4: The environment calculates and returns the reward value, and the agent obtains the temporal difference error. :

[0049]

[0050] Using timing difference error Calculate the gradient:

[0051]

[0052] Update online network parameters using stochastic gradient descent:

[0053]

[0054] in, This is the learning rate.

[0055] Step 4.4 proposes a progressive training strategy that uses a path length grading mechanism and dynamic dual-mode sampling rules to gradually adapt the training difficulty. The training process is carried out in stages, and a training sample set is generated through path length threshold constraints to reduce the dimensionality of early exploration.

[0056] The phase switching condition is: when consecutive Average reward obtained from rounds of training Exceeding the set threshold If the current strategy is deemed to be nearing saturation, a phase upgrade is triggered. Based on the current environmental data, new road data is added, and the system is improved. Exploring Greedy Strategies This is to meet the need to explore new entry paths.

[0057] Each training phase includes a dual-mode collaborative training approach of exploration and consolidation:

[0058] Targeted exploration mode: based on probability Targeted sampling of newly added path data in the current stage improves the agent's adaptability to new path data.

[0059] Hybrid consolidation mode: based on probability The system randomly selects learned paths from the historical path database and combines them with newly added paths for training, thereby enhancing the policy generalization ability across stages.

[0060] Step 1 requires acquiring road environment data from an open-source platform. Its core elements include nodes, paths, and relationships. Nodes are road intersections, paths are road segments connecting nodes, and relationships describe complex road topologies. These data structures are stored in the form of adjacency matrices and can be directly applied to path planning.

[0061] Step 2 involves preprocessing the road environment data to extract the actual latitude and longitude information of the nodes in the regional road network. In the actual processing, the latitude and longitude coordinates on the Earth's surface need to be converted into Cartesian coordinates by Mercator projection.

[0062] Meanwhile, in the road environment data, there are cases where one two-way separated road intersects with another two-way separated road. In this case, four nodes will be created. To simplify the road traffic network structure, all nodes located at the same intersection need to be merged. During this process, the tolerance needs to be carefully adjusted. If the tolerance is too small, it will not be able to cover all intersection nodes, but if the tolerance is too large, it will affect the setting of road nodes.

[0063] Furthermore, the present invention also provides an autonomous vehicle path planning system based on deep reinforcement learning, comprising:

[0064] Data acquisition module: used to acquire regional road environment data;

[0065] Preprocessing module: Performs preprocessing operations on the regional road environment data collected by the data acquisition module, including road node coordinate transformation and merging of adjacent road nodes;

[0066] Composite reward function module: Based on the actual needs of path planning for finding the shortest path, a composite reward function is designed;

[0067] Model building and training module: Based on the composite reward function, a path planning model is built using a deep reinforcement learning algorithm; the path planning model is trained using preprocessed regional road environment data to obtain a trained path planning model.

[0068] Furthermore, the present invention also provides an unmanned vehicle path planning device based on deep reinforcement learning, which includes an unmanned vehicle path planning method based on deep reinforcement learning.

[0069] The beneficial aspects of this invention are as follows:

[0070] (1) Most path planning research is limited to simulation verification in an idealized grid map environment, which is quite different from the real crisscrossing road network. Therefore, this paper will verify the feasibility of applying deep reinforcement learning to the real road environment based on the topological road environment data obtained from the open source platform, laying the foundation for promoting the transformation of autonomous driving from laboratory research to the market.

[0071] (2) An improved deep reinforcement learning algorithm is proposed, a composite reward function is designed, a neural network structure is built, and a progressive training strategy is proposed. Compared with the classic deep reinforcement learning algorithms DQN and Double DQN, the algorithm proposed in this paper has significantly improved the path optimality index, and all test paths can find effective paths. Attached Figure Description

[0072] Figure 1 This is a block diagram of a road path planning system.

[0073] Table 1 shows the path generation efficiency and path quality of different algorithms. Detailed Implementation

[0074] The present invention will now be further described with reference to the accompanying drawings.

[0075] refer to Figure 1 This paper details the road path planning system block diagram. First, it requires acquiring regional road environment data and preprocessing it. Latitude and longitude coordinates are converted to xy-plane coordinates using Mercator projection, and adjacent road nodes are merged, primarily to handle intersections and other intersecting roads, simplifying road nodes. Considering the actual needs of path planning, a composite reward function is designed, and a path planning model is built based on a deep reinforcement learning algorithm. The model is trained using the preprocessed road data, and the trained path planning model is saved. When needed, the user only needs to provide the starting point and the target point, and the model can provide the path.

[0076] Example:

[0077] Referring to Table 1, this invention uses road data to simulate and verify the proposed IMD3QN algorithm, which contains 702 paths. Since the action space selected in this invention is discrete, the classic deep reinforcement learning algorithms DQN and Double DQN are chosen as test algorithms for comparison. Path planning models obtained from different deep reinforcement learning algorithms are tested and compared. The path length output by the model is recorded by looping the starting point and the target point, and compared with the actual optimal path. The IMD3QN achieves a significant improvement in the number of optimal paths, with 562 test paths reaching the optimal level, a goodness of fit (optimal path percentage) of 80.1%, and all test paths finding valid paths. This represents a significant improvement compared to the classic deep reinforcement learning algorithms DQN and Double DQN.

[0078] Table 1 Comparison of path quality obtained from different algorithms

[0079] algorithm Number of optimal paths Excellence Number of valid paths not found IMD3QN 562 80.1% 0 Double DQN 476 67.8% 17 DQN 472 67.2% 8

[0080] In summary, applying the IMD3QN algorithm proposed in this paper to research on real road networks is of practical significance.

Claims

1. A path planning method for autonomous vehicles based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain regional road environment data; Step 2: Perform preprocessing operations on the regional road environment data, including road node coordinate transformation and merging of adjacent road nodes; Step 3: Consider the actual needs of shortest path planning and design a composite reward function; Step 4: Based on the composite reward function, a path planning model is constructed using a deep reinforcement learning algorithm; the path planning model is trained using preprocessed regional road environment data to obtain the trained path planning model; The composite reward function design in step 3 consists of three parts: path efficiency reward, exploration incentive mechanism, and loop inhibition mechanism. Combining these three parts yields the final composite reward function. ; Path efficiency reward: based on the path length obtained during training in the current round. Compared with the historical best path length The ratio is generated when the current path length is detected. Less than the historical best path length At that time, the historical best path update mechanism is triggered. And through preset performance reward factors Adjust path efficiency reward ; ； in, This represents the path length for training in the current round. The historical best path length for the same starting and ending points. As a performance-based reward factor; Explore incentive mechanisms: By judging the current time step Whether the agent accesses a new node and dynamically generates rewards ; ； in, Used to determine time steps Do you want to access a new node? For the exploration gain coefficient of dynamic attenuation, set: ； in, and It is a constant value, and in the initial stage, due to the time step The gain is relatively small, so a high gain coefficient is set to encourage breadth-first search. As the training progresses, Exponential decay gradually reduces the gain coefficient, guiding the strategy to migrate towards depth-first search. Cycle inhibition mechanism: Path looping is detected by a sliding time window. When a trajectory loop is detected within the time window, a fixed penalty coefficient is applied to generate a negative reward. ; ； in, For time window, This is a function indicating the repetition of the trajectory within a time window. It is a fixed penalty coefficient.

2. The unmanned vehicle path planning method based on deep reinforcement learning according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Define the state space and action space of the deep reinforcement learning algorithm; Step 4.2: Construct the neural network architecture for the deep reinforcement learning algorithm; Step 4.3: Construct the mathematical framework for deep reinforcement learning algorithms based on Markov decision processes; Step 4.4: Design a progressive training strategy to train the path planning model using preprocessed road data.

3. The unmanned vehicle path planning method based on deep reinforcement learning according to claim 2, characterized in that, The state space S in step 4.1 is a six-dimensional feature vector: ； in, Indicates the unit time step Multidimensional information within, and They are The normalized coordinates of the current node and the normalized coordinates of the target node at any given moment; express The normalized effective degree of the current node at time step 1 represents the number of walkable nodes adjacent to the current node, and is calculated as follows: ,in The maximum node degree in the road network; express The Euclidean distance between the current node and the target node at any given time; Action space It is a discrete decision set whose size dynamically depends on the effective degree of the current node. Each optional action For each feasible path branch originating from the current node, the total number of actions equals the effective degree of the current node. ,Right now .

4. The unmanned vehicle path planning method based on deep reinforcement learning according to claim 3, characterized in that, Step 4.2 establishes a neural network architecture, employing a multi-head attention-guided topology feature enhancement mechanism and a dual-network decoupling framework to achieve collaborative optimization of environmental perception and policy decision-making; by dynamically generating a node association weight matrix through a multi-head self-attention layer, topology features are weighted and fused to enhance graph structure perception capabilities; and by combining the dual-network architecture to decouple the policy selection and value evaluation process, collaborative optimization of environmental state perception and action decision-making is achieved. The network input is a six-dimensional feature vector, which is encoded into a high-dimensional feature vector through a fully connected layer. After being processed by a normalization layer, the input tensor is reconstructed into a 3D tensor and input to a four-head self-attention layer. A linear transformation is performed on the input tensor to generate a query vector, key vector, and value vector. Based on the query vector and key vector, the topological association strength matrix between nodes is calculated. The association weight matrix and the value vector are then weighted and summed to obtain the output feature Z. Z is then flattened and compared with the original high-dimensional feature vector. Perform residual connections to obtain features ; Construct independent value branches and action branches: the value branch outputs the state-value function. The value is mapped to a scalar value through a fully connected layer, representing the potential benefit of the current state in long-term path planning; Advantage branch output action advantage function Its dimension matches the action space, characterizing the relative advantage difference of each action in immediate decision-making; to avoid the absolute bias of the advantage value, the final Q value is generated through a decentralized aggregation rule, where Q(s,a) is a state-action value function, representing the long-term expected return of choosing action a in state s. The decentralized aggregation rule calculation process satisfies: ； in, Indicates the state Under these conditions, the average expected cumulative reward represents the intrinsic value of a state; Indicates the state Select action The relative advantage of satisfying the zero centralization constraint ; Let be the mean of the dominance function. The current state The effective number of actions below, used to maintain and Consistency is maintained to prevent gradient instability; Simultaneously, a dual-network architecture decoupling strategy selection and value assessment process is adopted: online network Real-time response to environmental conditions, generation of action strategies, target network A stable Q-value evaluation benchmark is provided through periodic parameter synchronization; during the training phase, the target Q-value is calculated through a dual-verification mechanism. : ； in, , It is a discount factor used to balance the weight of immediate rewards and long-term rewards. For online network output, This indicates that the optimal action is selected by the online network. For the output of the target network, This indicates that the Q-value of the action is evaluated by the target network; this decoupling mechanism allows the selection of the action with the maximum Q-value and the evaluation of the corresponding Q-value to be handled by the online network. and target network Perform the actions separately to suppress the systematic overestimation of Q values ​​caused by the function approximation error; Target Network parameters A delayed update mechanism is adopted, with updates at fixed intervals. Perform a hard update, online network Synchronize parameters to the target network ,Right now .

5. The unmanned vehicle path planning method based on deep reinforcement learning according to claim 4, characterized in that, Step 4.3, based on Markov decision processes, constructs the mathematical framework for a deep reinforcement learning model. The specific steps are as follows: Step 4.3.1: The agent observes the state in the environment. ; Step 4.3.2: Based on the current node's effective degree Generate a subset of executable actions ; Step 4.3.3: Based on the strategy generated by the deep neural network, the agent selects a subset of actions according to... A greedy strategy selects actions based on probability. Explore randomly, otherwise maximize. Actions; after an action is performed, the environment updates its state value. ; ； Step 4.3.4: The environment calculates and returns the reward value, and the agent obtains the temporal difference error. : 。 6. The unmanned vehicle path planning method based on deep reinforcement learning according to claim 5, characterized in that, Step 4.4 proposes a progressive training strategy, which uses a path length grading mechanism and dynamic dual-mode sampling rules to gradually adapt the training difficulty; the training process is carried out in stages, and a training sample set is generated by path length threshold constraints to reduce the dimensionality of early exploration. The phase switching condition is: when consecutive Average reward obtained from rounds of training Exceeding the set threshold If the current strategy is deemed to be nearing saturation, a phase upgrade is triggered. Based on the current environmental data, new road data is added, and the system is improved. Exploring Greedy Strategies To meet the need to explore new entry paths; Each training phase includes a dual-mode collaborative training approach of exploration and consolidation: Targeted exploration mode: based on probability Targeted sampling of newly added path data in the current stage improves the agent's adaptability to new path data; Hybrid consolidation mode: based on probability The system randomly selects learned paths from the historical path database and combines them with newly added paths for training, thereby enhancing the policy generalization ability across stages.

Images