A lane-changing trajectory planning method for autonomous vehicles based on a hybrid model of HDDQN and TCN

Abstract

The present invention discloses a lane-changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN. The specific method is as follows: constructing a lane-changing trajectory planning model for an autonomous driving vehicle; obtaining historical environmental spatial state information to train the autonomous driving lane-changing trajectory model; using the trained lane-changing trajectory planning model for an autonomous driving vehicle to predict the autonomous driving lane trajectory; Advantages: The present invention can adapt to the ever-changing complex traffic environment, has good stability, can cope with complex control problems, and provide technical support for the autonomous driving lane-changing system; using TCN to process data helps DDQN to better understand the relationship and dynamic changes between vehicles, thereby making more accurate target decisions and trajectory planning actions, and helps to improve the lane-changing efficiency and stability of autonomous driving; by improving the reward function, the vehicle lane-changing stability is ensured while improving the comfort of passengers.

Description

Technical Field

[0001] The present invention belongs to the field of trajectory planning for autonomous driving, and specifically relates to a lane-changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN. Background Art

[0002] At present, the lane-changing trajectory planning methods for autonomous vehicles can be divided into three categories: traditional trajectory planning algorithms, intelligent optimization algorithms, and trajectory planning algorithms based on reinforcement learning.

[0003] Traditional trajectory planning algorithms are simple in principle and easy to implement, and have been widely used in the field of path planning for autonomous vehicles. Common methods include curve interpolation, graph search, sampling-based methods, and numerical optimization algorithms. Patent (CN202410216710.1) discloses a personalized lane change trajectory planning method based on quintic polynomials, which increases the degree of personalization of lane change trajectories from "segmented" to "continuous." The proposed personalized lane change trajectory planning method is suitable for structured road lane change scenarios for autonomous or assisted driving, and the planned trajectory meets stability and collision avoidance requirements. Literature (Research on A* Algorithm for Static Path Planning Based on Neighborhood Expansion. Guo Xiaojing & Yang Zhuocheng. Computer Engineering and Applications, 2022, 58(8):168-174.DOI:10.3778 / j.issn.1002-8331.2010-0222.) In order to solve the problems of low search freedom of the traditional A* algorithm, long planned path length and large turning angle, an improved A* algorithm is proposed. The improved algorithm expands the traditional 8-neighborhood search to 24 neighborhoods, and uses the guide vector to optimize the number of neighborhoods to improve the search efficiency; the path smoothing algorithm is used to eliminate redundant nodes in the path and optimize the smooth path.

[0004] While traditional trajectory planning algorithms offer significant advantages, they often suffer from high computational complexity, extreme reliance on environmental information, and poor real-time performance. The rise and rapid development of artificial intelligence has led to the emergence of various biomimetic intelligent optimization algorithms in trajectory planning, such as ant colony algorithms, genetic algorithms, neural network algorithms, and particle swarm algorithms. Patent (CN202011340491.6) discloses a safe autonomous driving lane change trajectory planning method based on an improved LSTM neural network. This method uses a safety-focused trajectory algorithm to generate a safe trajectory, then uses an improved LSTM neural network to learn the safe trajectory from the original trajectory. This ensures that the final lane change trajectory is closer to the human riding experience while maintaining safety.

[0005] Intelligent optimization algorithms have good adaptability and improve real-time performance compared to traditional planning algorithms. However, they cannot converge quickly, and their ability to adapt to complex environments needs to be improved. They cannot guarantee stability and optimal solutions. Therefore, research on path planning based on reinforcement learning algorithms has become increasingly intense. The paper (Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning.KB Naveed,Z. Qiao and JM Dolan.2021IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA, 2021, pp.601-606) proposed a hierarchical reinforcement learning structure combined with a proportional-integral-derivative (PID) controller for trajectory planning. The results show that this framework reduces convergence time and helps cars learn in more dynamic environments.

[0006] Patent (CN202310347395.1) describes a lane-changing method for autonomous vehicles based on layered deep reinforcement learning. This method uses a deep reinforcement learning DQN network and a DDPG network for lane-changing decisions and trajectory planning and training, respectively. Current reinforcement learning-based methods can adapt to changing traffic conditions, but they suffer from low stability and need further improvement in trajectory planning accuracy. Summary of the Invention

[0007] In order to improve the stability of autonomous vehicles during lane changes and improve the accuracy of trajectory planning during lane changes, this paper proposes a lane change trajectory planning method for autonomous vehicles based on a hybrid model of HDDQN and TCN. This method can realize lane change decision-making and trajectory planning for autonomous driving in complex environments, thereby improving the lane change efficiency and stability of autonomous driving.

[0008] The technical solutions adopted are:

[0009] A lane-changing trajectory planning method for an autonomous vehicle based on a hybrid model of HDDQN and TCN is characterized by the following specific methods:

[0010] S1. Construct a lane-changing trajectory planning model for an autonomous vehicle. The lane-changing trajectory planning model includes an input layer, a lane-changing layer, and an output layer. The lane-changing layer is composed of a temporal convolutional network (TCN) and two dual-depth Q networks (DDQNs), where:

[0011] The TCN and one DDQN form the lane-changing decision layer, and the TCN and another DDQN form the trajectory planning layer. The environment state space information inputted by the input layer passes through the TCN and is then inputted into the two DDQNs respectively. The lane-changing decision target outputted by the DDQN in the lane-changing decision layer is inputted into the DDQN in the trajectory planning layer. The predicted trajectory planning action outputted by the DDQN in the trajectory planning layer is outputted via the output layer.

[0012] S2. Obtain historical environmental spatial state information and use it as a training set to train the autonomous driving lane change trajectory model;

[0013] S3. Use the trained autonomous vehicle lane change trajectory planning model to predict the autonomous vehicle lane trajectory.

[0014] In a further preferred embodiment of the technical solution of the present invention, the environment state space information includes at least the coordinates, speeds and lane IDs of the target lane-changing vehicle and the preceding vehicle in the current lane, the preceding vehicle in the target lane and the following vehicle in the target lane.

[0015] Further preference is given to the technical solution of the present invention, TCN includes a cascaded input layer, N residual blocks, a fully connected layer and an output layer, each residual block includes a 1x1 convolution block and a first dilated causal convolution layer, a first weight normalization layer, a first ReLU activation function, a first Dropout layer, a second dilated causal convolution layer, a second weight normalization layer, a second ReLU activation function, and a second Dropout layer connected in sequence, and the output of the nth residual block is respectively input into the 1x1 convolution block in the n+1th residual block to adjust the number of channels and the first dilated causal convolution layer, and the output of the 1x1 convolution block in the n+1th residual block is superimposed with the output of the second Dropout layer as the input of the n+2th residual block.

[0016] In a further optimization of the technical solution of the present invention, the reward R of the DDQN in the lane change decision layer is g Reward R by safety safe and efficiency reward R eff2 Composition, characterized by:

[0017] R g =R safe +R eff2 ;

[0018] The reward R of the trajectory planning layer a Reward R by safety safe , efficiency reward R eff and smoothness reward R smooth Composition, characterized by:

[0019] R a =R safe +Reff +R smooth ;

[0020] Safety Reward R safe Including collision reward R safe1 and unsafe rewards and penalties R safe2 , characterized by:

[0021] R safe =R safe1 +R safe2 ;

[0022] When the target lane-changing vehicle is in the current lane:

[0023]

[0024] When the target lane-changing vehicle is in the target lane:

[0025]

[0026] Where D is the vehicle length, σ1 is the reward constant for collision reward and punishment, Δx CPV is the distance between the target lane-changing vehicle and the preceding vehicle in the current lane, Δx TFV is the distance between the target lane-changing vehicle and the preceding vehicle in the target lane, Δx TPV is the distance between the target lane-changing vehicle and the vehicle behind it in the target lane;

[0027]

[0028] Where Δx r is the expected safe distance between the target lane-changing vehicle and the preceding vehicle in the current lane. w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following and lane-changing phases, respectively. σ2 is the reward constant for the unsafe rewards and penalties. Carfollowing represents the following phase, and Lane changing represents the lane-changing phase.

[0029] Efficiency Reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​limit rewards and penalties R eff2 and the reward and penalty R for not reaching the lane change target range eff3 , characterized by:

[0030] R eff =R eff1 +R eff2 +R eff3

[0031]

[0032] Where w3 is the weight parameter of lane change timeout reward and punishment, σ3 is the reward constant of lane change timeout reward and punishment, Δt is the actual lane change time, and Δtr is the expected lane change time;

[0033]

[0034] Where, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and punishment and the minimum speed limit reward and punishment respectively, σ4 is the reward constant of the speed limit reward and punishment, v SV is the target lane-changing vehicle speed;

[0035]

[0036] Where x r1 and x r2 Set the upper and lower limits of the expected range for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range;

[0037] Smoothness reward R smooth Characterized by:

[0038] R smooth =-w6|a t+1 -a t |-w7|k t+1 -k t |

[0039] Where w6 and w7 are the weight parameters corresponding to acceleration reward and curvature reward, respectively. t+1 and a t are the vehicle accelerations at time t+1 and t, respectively, k t+1 and k t are the curvatures of the lane-changing trajectory at time t+1 and time t, respectively.

[0040] In a further embodiment of the technical solution of the present invention, during the training of the automatic driving lane change trajectory model:

[0041] The experience replay pool of the lane change decision layer stores information including the state s at time t t , the historical environment state space information h at time t t , the lane change decision target g at time t t , the reward R of DDQN in the lane-changing decision layer at time t gt , the state s at time t+1 t+1 And the historical environment state space information h at time t+1 t+1 ;

[0042] The experience replay pool of the trajectory planning layer stores information including the state s at time t t, the historical environment state space information h at time t t , trajectory planning action a at time t t , the reward R of DDQN in the trajectory planning layer at time t at , the state s at time t+1 t+1 , the historical environment state space information h at time t+1 t+1 And whether the lane change action is completed t .

[0043] In a further optimization of the technical solution of the present invention, during the training of the automatic driving lane change trajectory model:

[0044] For the lane-changing decision layer, the value function of the target network is:

[0045]

[0046] The loss function is:

[0047]

[0048] The current network parameters are updated as follows:

[0049]

[0050] Where E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer at time t, γ∈[0,1] is the discount factor, α is the learning rate, and R gt is the reward of DDQN in the lane-changing decision layer at time t, g is the set of all possible lane-changing vehicles, s t is the state at time t, s t+1 is the state at time t+1;

[0051] For the trajectory planning layer, the value function of the target network is:

[0052]

[0053] The loss function is:

[0054]

[0055] The current network parameters are updated as follows:

[0056]

[0057] Where R at is the reward of DDQN in the trajectory planning layer at time t, a is the set of all possible trajectory planning actions, and are the current network parameters and target network parameters of the trajectory planning layer at time t, respectively.

[0058] In a further optimization of the technical solution of the present invention, during the training process of the automatic driving lane changing trajectory model: a greedy strategy is used to select the lane changing decision target and trajectory planning action.

[0059] In a further optimization of the technical solution of the present invention, during the training process of the automatic driving lane changing trajectory model: with improving the lane changing success rate as the optimization goal, multiple rounds of training are performed to obtain the automatic driving lane changing trajectory model parameters with the optimal lane changing success rate.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The lane-changing trajectory planning method for autonomous vehicles based on the hierarchical DDQN and TCN hybrid model can adapt to the ever-changing and complex traffic environment, has good stability, can cope with complex control problems, and provide technical support for autonomous driving lane-changing systems.

[0062] 2. The lane change decision model in the lane change decision layer and the trajectory planning model in the trajectory planning layer of the present invention both use TCN to process data, which helps DDQN better understand the relationship and dynamic changes between vehicles, thereby making more accurate target decisions and trajectory planning actions, and helping to improve the lane change efficiency and stability of autonomous driving.

[0063] 3. The present invention improves the reward function, which ensures the stability of vehicle lane change while improving the comfort of passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a lane-changing trajectory planning model architecture for autonomous vehicles based on a hybrid model of layered dual-depth Q-network and temporal convolutional network;

[0065] Figure 2(a) is a schematic diagram of causal expansion convolution;

[0066] Figure 2(b) TCN residual block;

[0067] Figure 2(c) TCN data processing steps;

[0068] Figure 3 It is a training process for autonomous vehicle lane change trajectory planning based on a hybrid model of layered dual-depth Q-network and temporal convolutional network;

[0069] Figure 4 This is a schematic diagram of the lane changing process. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following Figure 1-4, the present invention is described in further detail.

[0071] It should be understood that the description herein is only used to explain the present invention and is not intended to limit the present invention.

[0072] The technical solution adopted by the present invention is a lane-changing trajectory planning method for an autonomous vehicle based on a hybrid model of a layered dual-depth Q network and a temporal convolutional network. The method includes the following two steps:

[0073] Step A. Construction of a lane-changing trajectory planning model for autonomous vehicles based on the HDDQN+TCN hybrid model;

[0074] Step B. Training and testing of the lane-changing trajectory planning model for the autonomous vehicle.

[0075] The lane-changing trajectory planning model for the autonomous vehicle constructed in step A includes an input layer, a lane-changing layer, and an output layer. Figure 1 shown.

[0076] A.1 Input layer: The input of the lane-changing trajectory planning model for autonomous vehicles is the historical environment state space information h t , by the state s at multiple moments t Composition, that is, h t ={s1,s2…s t The state s specifically includes the coordinates, speed, and lane ID of the target lane-changing vehicle and surrounding vehicles, and can be represented as follows: s = [lane IDSV ,lane IDCPV ,lane IDTPV ,lane IDTFV ,v SV ,v CPV ,v TPV ,v TFV ,Δx CPV ,Δx TPV ,Δx TFV

[0077] Where lane IDSV The lane ID of the target lane-changing vehicle;

[0078] lane IDCPV 、lane IDTPV and lane IDTFV They are the lane IDs of the vehicle in front of the current lane, the vehicle in front of the target lane, and the vehicle behind the target lane;

[0079] v SV is the target lane-changing vehicle speed;

[0080] v CPV 、v TPV and v TFVare the speeds of the front vehicle in the current lane, the front vehicle in the target lane, and the rear vehicle in the target lane respectively;

[0081] Δx CPV =|x CPV -x SV | is the distance between the target lane-changing vehicle and the preceding vehicle in the current lane, x SV and x CPV are the position coordinates of the center of the target lane-changing vehicle and the center of the vehicle in front of the current lane along the lane direction respectively;

[0082] Δx TPV =|x TPV -x SV | is the distance between the target lane-changing vehicle and the preceding vehicle in the target lane, x TPV is the position coordinate of the center of the front vehicle in the target lane along the lane direction;

[0083] Δx TFV =|x SV -x TFV |Distance between the target lane-changing vehicle and the vehicle behind it in the target lane, x TFV is the position coordinate of the center of the vehicle behind the target lane along the lane direction.

[0084] A.2 Lane-changing layer: The lane-changing layer structure includes a lane-changing decision layer and a trajectory planning layer. Each layer adopts a DDQN and TCN architecture. The lane-changing decision layer uses the historical environment state space information and the reward R g The lane-changing decision model is trained to fit the value function Q, and the current network is used to select a target to maximize the current network value. The target lane-changing vehicle selects the lane-changing decision target with a probability of (1-ε) according to the greedy strategy.

[0085] Among them, the trajectory planning layer uses the historical environment state space information, lane change decision goals and reward R a The trajectory planning model is trained and fitted, and the current network is used to select a trajectory planning action to maximize the current network value. The target lane-changing vehicle selects a lane-changing trajectory planning action according to the greedy strategy with a probability of (1-ε).

[0086] The TCN component of the layer is the core feature extractor for the entire lane-changing layer. It captures the long-term dependencies of the historical environmental state space information in the input layer. This allows it to process noisy or inaccurate data, improve the model's generalization capabilities, and enhance the accuracy of subsequent lane-changing decisions and trajectory planning. This is achieved through the following steps, as shown in Figure 2(c):

[0087] First, the input of TCN is the historical environment state space information h t .

[0088] Secondly, the data filling method in the TCN data processing process is set to causal dilation convolution. Specifically, the neuron output of the TCN at time t is only related to s1, s2…s t Related to the future state t+1 ,s t+2 ,s t+3 …s t+n And by sampling the TCN input data exponentially layer by layer, a larger receptive field can be obtained with fewer network layers, as shown in Figure 2(a). The specific calculation method is represented by the following formula:

[0089]

[0090] Where F(s) is the convolution operation on the sequence element s,

[0091] f is the number of filters,

[0092] k is the size of the convolution kernel,

[0093] d is the expansion factor,

[0094] x s-d·i To perform convolution only on the past state,

[0095] * is the convolution operation.

[0096] At this point, the receptive field can be characterized as:

[0097] field=(k-1)·d+1

[0098] Furthermore, by increasing the network depth and replacing convolutional layers with residual blocks, we can mitigate the vanishing or exploding gradient problems associated with increased network depth. Residual links are used to transfer information across layers within the network. The structure of the residual block is shown in Figure 2(b). It features two layers of dilated causal convolutions and ReLu activation functions. Weight normalization layers and Dropout are also added to each layer to regularize the network and improve its generalization capabilities.

[0099] Furthermore, the TCN network parameters are set, including the input layer shape, output layer dimension, number of filters, convolution kernel size and other parameters, as shown in Table 1 below:

[0100] Table 1 TCN network parameters

[0101]

[0102] Finally, the TCN output is an abstract representation of the characteristics and temporal relationships of the input information, capturing its temporal evolution and changes, and understanding its temporal patterns and structures. This subsequently serves as input to the lane change decision layer and trajectory planning layer.

[0103] For the reward R in the DDQN structure, there are three types: safety reward, efficiency reward, and smoothness reward.

[0104] Among them, the reward R of the lane-changing decision layer is g Reward R by safety safe and efficiency rewards (only speed limit rewards and penalties) R eff2 Composition, characterized by the following formula:

[0105] R g =R safe +R eff2

[0106] Among them, the reward R of the trajectory planning layer a Reward R by safety safe , efficiency reward R eff and smoothness reward R smooth Composition, characterized by the following formula:

[0107] R a =R safe +R eff +R smooth

[0108] The three types of rewards are obtained in the following ways:

[0109] (1) Safety reward R safe Including collision reward R safe1 and unsafe rewards and penalties R safe2 :

[0110] R safe =R safe1 +R safe2

[0111] The collision reward and penalty uses the vehicle center point as the coordinate point. During the following or lane-changing phase, if the distance between the target lane-changing vehicle and the preceding or following vehicle in the same lane is less than or equal to the vehicle length D, it is considered a collision and the corresponding penalty is given. Otherwise, the corresponding reward is given. It is represented by the following formula:

[0112] When the target lane-changing vehicle is in the current lane:

[0113]

[0114] When the target lane-changing vehicle is in the target lane:

[0115]

[0116] Where D is the vehicle length (m), and σ1 is the reward constant for collision reward and punishment.

[0117] Among them, unsafe rewards and penalties, the target lane-changing vehicle needs to maintain a certain expected safe distance from the vehicle in front of it in the current lane during the following or lane-changing process to ensure vehicle safety and respond to emergencies. It is represented by the following formula:

[0118]

[0119] Where Δx r is the expected safe distance between the target lane-changing vehicle and the preceding vehicle in the current lane, w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following stage and the lane-changing stage, respectively, and σ2 is the reward constant for the unsafe rewards and penalties.

[0120] (2) Efficiency reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​limit rewards and penalties R eff2 and the reward and penalty R for not reaching the lane change target range eff3 , represented by the following formula:

[0121] R eff =R eff1 +R eff2 +R eff3

[0122] Among them, the lane change timeout reward and penalty, when the lane change decision layer outputs the option of lane change, sets the expected lane change time for the target lane change vehicle to ensure the lane change efficiency, which is represented by the following formula:

[0123]

[0124] Where w3 is the weight parameter of lane change timeout reward and punishment, σ3 is the reward constant of lane change timeout reward and punishment, Δt is the actual lane change time, and Δt r is the expected lane change time.

[0125] Among them, the speed limit reward and punishment, when the target lane-changing vehicle is driving, it is necessary to set a speed limit for the target lane-changing vehicle to improve the traffic efficiency of the road while ensuring safety. It is represented by the following formula:

[0126]

[0127] Where, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and punishment and the minimum speed limit reward and punishment respectively, and σ4 is the reward constant of the speed limit reward and punishment.

[0128] Among them, the reward and penalty for not reaching the lane change target range. When the lane change decision layer outputs the option of lane change, the expected range is set for the final lane change point of the target lane change vehicle to ensure that the vehicle selects the appropriate lane change trajectory and improve the lane change efficiency. It is represented by the following formula:

[0129]

[0130] Where x r1 and x r2 The upper and lower limits of the expected range are set for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range.

[0131] (3) Smoothness reward R smooth In addition to ensuring safety and efficiency, autonomous vehicles also need to provide drivers and passengers with a good riding experience. Acceleration a and lane change trajectory curvature k are used as constraints to ensure comfort during vehicle driving, which can be expressed as follows:

[0132] R smooth =-w6|a t+1 -a t |-w7|k t+1 -k t |

[0133] Where w6 and w7 are the weight parameters corresponding to acceleration reward and penalty and curvature reward and penalty respectively.

[0134] A.3 Output Layer: The lane change decision layer in the lane change layer can output the lane change decision target, including lane change or car following; the trajectory planning layer can output the predicted trajectory planning action, including trajectory coordinates and the corresponding coordinate speed.

[0135] Among them, step B includes three parts: autonomous driving vehicle lane change trajectory planning model training process, training data set generation, and model optimization.

[0136] B. In the lane-changing trajectory planning model for autonomous vehicles, parameters include historical environment state space information, lane-changing decision target set, trajectory planning action set, and rewards. Adjustable parameters include the discount factor, exploration rate, number of TCN convolutional layers, convolution kernel size, number of filters, and dilation coefficient.

[0137] First, the networks of the lane change decision layer and trajectory planning layer are initialized with random current network parameters θ g and θ a Initialize the current network and copy it to the target network. The lane change decision layer and trajectory planning layer each have an experience replay pool D1 and D2. Initialize the experience replay pools D1 and D2, set the maximum capacity of the replay memory, set the number of training times, and set the time limit for each round.

[0138] Each round of training consists of several episodes, and each episode consists of several time steps.

[0139] In each round, the historical environment state space information is initialized, the TCN is used to process the historical environment state space information, and the relevant variables are initialized. The relevant variables include collision history records, rewards, and other related information generated during the execution of the action.

[0140] At each time step, the target lane-changing vehicle selects the lane-changing decision target and trajectory planning action based on the historical environment state space information.

[0141] For the lane-changing decision layer, based on the processed historical environment state space information, Q(s, g, θ g ) to approximate the value function Q of the optimal decision target * (s,g). Then use the current network and use the state s t+1 Select a goal to maximize the current network value. And the target lane-changing vehicle selects the optimal lane-changing decision target with a probability of (1-ε) according to the greedy strategy. It can be specifically characterized as:

[0142]

[0143] In the formula, g is the set of all possible targets, The current network is in state s t+1 The objective value function under .

[0144] For the trajectory planning layer, based on the target output by the lane change decision layer and the processed historical environment state space information, Q(s, a; g, θ a ) to approximate the value function Q of the optimal trajectory planning action * (s,a;g). Then use the current network and use the state s t+1 Select an action to maximize the current network value. And use the greedy strategy to select the optimal trajectory planning action with a probability of (1-ε). It can be specifically characterized as:

[0145]

[0146] Where a is the set of all possible actions, The current network is in state s t+1 The action-value function under .

[0147] Execute the action and get the new state s t+1 , the reward R of the lane-changing decision layer g , the reward R of the trajectory planning layer a and other information.

[0148] Store the current state s in the experience replay pool D1 t , historical environment state space information h t , lane change decision target g t , Lane change decision layer reward R gt , new state s t+1 , new historical environment state space information h t+1 etc.; store the current state s in the experience replay pool D2 t , historical environment state space information h t , trajectory planning action a t , trajectory planning layer reward R at , new state s t+1 , new historical environment state space information h t+1 And whether the action is completed.

[0149] At each time step, a small batch of samples D is randomly drawn from the experience replay pool to calculate the lane change decision layer value function and trajectory planning layer value function The target value is compared with the current network value function to calculate the loss function of the lane change decision layer and the trajectory planning layer respectively. and Update the current network parameters using gradient descent method and

[0150] For the lane-changing decision layer, the value function of the target network is:

[0151]

[0152] The loss function is obtained as:

[0153]

[0154] Then the current network parameters are updated as follows:

[0155]

[0156] Where E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer, respectively. The former is used to select the target (the strategy of the current state), and the latter is used to evaluate the value of the current target. γ∈[0,1] is the discount factor, and α is the learning rate.

[0157] For the trajectory planning layer, the value function of the target network is:

[0158]

[0159] The loss function is obtained as:

[0160]

[0161] The current network parameters are updated as follows:

[0162]

[0163] Where, and They are the current network parameters and target network parameters of the trajectory planning layer. The former is used to select actions (the strategy of the current state), and the latter is used to evaluate the value of the current action.

[0164] Update the target network parameters θ every several time steps g′ =θ g ,θ a′ =θ a .

[0165] The above process is repeated, and the round ends when the vehicle collides, the set time limit is reached, or the autonomous vehicle completes the lane change.

[0166] Repeat the above steps and continue training until all rounds in a round of training are completed, and the first round of training ends.

[0167] With the optimization goal of improving the lane-changing success rate, multiple rounds of training are conducted, and the lane-changing model parameters are continuously adjusted based on the training results to determine the model parameters that can optimize the lane-changing success rate.

[0168] B.2 Training Dataset Generation: This dataset primarily contains real-time motion information of the target vehicle and surrounding vehicles during the lane change process, including coordinates, lane ID, and speed. The lane change process in this real-time motion information is calibrated by changes in the horizontal coordinates of the target vehicle.

[0169] The lane-changing process is divided into the lane-changing decision stage (e.g. Figure 4 AB phase), lane change execution phase (such as Figure 4 BC phase) and lane change completion phase (e.g. Figure 4 The lane change decision phase takes the historical environment state space information as input, corresponds to the lane change decision layer in step A, and outputs the lane change decision target. The lane change execution phase and lane change completion phase correspond to the trajectory planning layer in step A, and output the trajectory planning action.

[0170] B.3 Optimization of the lane-changing trajectory planning model for autonomous vehicles: The model is trained using a training dataset. Based on the training results, such as the loss function value and lane-changing success rate, the model parameters are adjusted, such as the number of TCN convolutional layers, convolution kernel size, number of filters, dilation coefficient, discount factor, and exploration rate. The model is repeatedly trained under different parameters to determine the parameters that achieve the optimal model training results.

[0171] This paper proposes an improved hierarchical lane-changing trajectory planning model based on the temporal convolutional network (TCN). Dual deep Q networks are used for training and testing of the lane-changing decision layer and the trajectory planning layer. Each layer uses TCN to process data, which helps the DDQN model better understand the relationship and dynamic changes between vehicles, thereby making more accurate decisions and planning, and helping to improve the lane-changing efficiency and stability of autonomous driving.

[0172] The parts not involved in the present invention are the same as the existing technology or can be implemented by using the existing technology.

[0173] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A lane-changing trajectory planning method for an autonomous vehicle based on a hybrid model of HDDQN and TCN, characterized by: The specific method is as follows: S1. Construct a lane-changing trajectory planning model for an autonomous vehicle. The lane-changing trajectory planning model includes an input layer, a lane-changing layer, and an output layer. The lane-changing layer is composed of a temporal convolutional network (TCN) and two dual-depth Q networks (DDQNs), where: The TCN and one DDQN form the lane-changing decision layer, and the TCN and another DDQN form the trajectory planning layer. The environment state space information inputted by the input layer passes through the TCN and is then inputted into the two DDQNs respectively. The lane-changing decision target outputted by the DDQN in the lane-changing decision layer is inputted into the DDQN in the trajectory planning layer. The predicted trajectory planning action outputted by the DDQN in the trajectory planning layer is outputted via the output layer. S2. Obtain historical environmental spatial state information and use it as a training set to train the autonomous driving lane change trajectory model; S3. Use the trained autonomous vehicle lane change trajectory planning model to predict the autonomous vehicle lane trajectory.

2. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: The environment state space information includes at least the coordinates, speeds, and lane IDs of the target lane-changing vehicle and the preceding vehicle in the current lane, the preceding vehicle in the target lane, and the following vehicle in the target lane.

3. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: TCN includes a cascaded input layer, N residual blocks, a fully connected layer and an output layer. Each residual block includes a 1x1 convolution block and the first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function, the first Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function, and the second Dropout layer. The output of the nth residual block is respectively input into the 1x1 convolution block in the n+1th residual block to adjust the number of channels and the first dilated causal convolution layer. The output of the 1x1 convolution block in the n+1th residual block is superimposed with the output of the second Dropout layer as the input of the n+2th residual block.

4. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: The reward R of DDQN in the lane-changing decision layer g Reward R by safety safe and efficiency reward R eff2 Composition, characterized by: R g =R safe +R eff2 ; The reward R of the trajectory planning layer a Reward R by safety safe , efficiency reward R eff and smoothness reward E smooth Composition, characterized by: R a =R safe +R eff +R smooth ; Safety Reward R safe Including collision reward R safe1 and unsafe rewards and penalties R safe2 , characterized by: R safe =R safe1 +R safe2 ; When the target lane-changing vehicle is in the current lane: When the target lane-changing vehicle is in the target lane: Where D is the vehicle length, σ1 is the reward constant for collision reward and punishment, Δx CPV is the distance between the target lane-changing vehicle and the preceding vehicle in the current lane, Δx TFV is the distance between the target lane-changing vehicle and the preceding vehicle in the target lane, Δx TPV is the distance between the target lane-changing vehicle and the vehicle behind it in the target lane; Where Δx r is the expected safe distance between the target lane-changing vehicle and the preceding vehicle in the current lane. w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following and lane-changing phases, respectively. σ2 is the reward constant for the unsafe rewards and penalties. Carfollowing represents the following phase, and Lane changing represents the lane-changing phase. Efficiency Reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​limit rewards and penalties R eff2 and the reward and penalty R for not reaching the lane change target range eff3 , characterized by: R eff =R eff1 +R eff2 +R eff3 Where w3 is the weight parameter of lane change timeout reward and punishment, σ3 is the reward constant of lane change timeout reward and punishment, Δt is the actual lane change time, and Δt r is the expected lane change time; Where, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and punishment and the minimum speed limit reward and punishment respectively, σ4 is the reward constant of the speed limit reward and punishment, v SV is the target lane-changing vehicle speed; Where x r1 and x r2 Set the upper and lower limits of the expected range for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range; Smoothness reward R smooth Characterized by: <h2 style=";text-align:left;direction:ltr">R<h2 style=";text-align:left;direction:ltr"> smooth <h2 style=";text-align:left;direction:ltr"> =-w6|a<h2 style=";text-align:left;direction:ltr"> t+1 <h2 style=";text-align:left;direction:ltr"> -a<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> |-w7|k<h2 style=";text-align:left;direction:ltr"> t+1 <h2 style=";text-align:left;direction:ltr"> -k<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> | Where w6 and w7 are the weight parameters corresponding to acceleration reward and curvature reward, respectively. +1 and a t are the vehicle accelerations at time t+1 and t, respectively, k t+1 and k t are the curvatures of the lane-changing trajectory at time t+1 and time t, respectively.

5. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: The experience replay pool of the lane change decision layer stores information including the state s at time t t , the historical environment state space information h at time t t , the lane change decision target g at time t t , the reward R of DDQN in the lane-changing decision layer at time t gt , the state s at time t+1 t+1 And the historical environment state space information h at time t+1 t+1 ; The experience replay pool of the trajectory planning layer stores information including the state s at time t t , the historical environment state space information h at time t r , trajectory planning action a at time t t , the reward R of DDQN in the trajectory planning layer at time t at , the state s at time t+1 t+1 , the historical environment state space information h at time t+1 t+1 And whether the lane change action is completed t .

6. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: For the lane-changing decision layer, the value function of the target network is: The loss function is: The current network parameters are updated as follows: Where E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer at time t, γ∈[0,1] is the discount factor, α is the learning rate, and R gt is the reward of DDQN in the lane-changing decision layer at time t, g is the set of all possible lane-changing vehicles, s t is the state at time t, s t+1 is the state at time t+1; For the trajectory planning layer, the value function of the target network is: The loss function is: The current network parameters are updated as follows: Where R at is the reward of DDQN in the trajectory planning layer at time t, a is the set of all possible trajectory planning actions, and are the current network parameters and target network parameters of the trajectory planning layer at time t, respectively.

7. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: A greedy strategy is used to select lane-changing decision targets and trajectory planning actions.

8. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: With the optimization goal of improving the lane change success rate, multiple rounds of training are conducted to obtain the parameters of the autonomous driving lane change trajectory model with the optimal lane change success rate.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 8 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images