Automatic driving vehicle lane changing trajectory planning method based on HDDQN and TCN hybrid model

Abstract

The invention discloses an automatic driving vehicle lane changing track planning method based on an HDDQN and TCN hybrid model. The method specifically comprises the following steps: constructing an automatic driving vehicle lane changing track planning model; acquiring historical environment space state information to train an automatic driving lane changing trajectory model; predicting an automatic driving lane track by adopting the trained automatic driving vehicle lane changing track planning model; the method has the advantages that the method can adapt to constantly changing complex traffic environments, is good in stability, can deal with complex control problems, and provides technical support for an automatic driving lane changing system; the TCN is used for processing the data, so that the DDQN is helped to better understand the relationship and dynamic change between the vehicles so as to make more accurate target decision and trajectory planning actions, and the lane changing efficiency and stability of automatic driving are helped to be improved; the award function is improved, so that the comfort of passengers is improved while the lane changing stability of the vehicle is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of trajectory planning for autonomous driving, and specifically relates to a lane changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN. Background Art

[0002] At present, the lane change trajectory planning methods for autonomous driving vehicles can be divided into three categories: traditional trajectory planning algorithms, intelligent optimization algorithms, and trajectory planning algorithms based on reinforcement learning.

[0003] The traditional trajectory planning algorithm is simple in principle and easy to implement, and has been widely used in the field of path planning for autonomous vehicles. Common methods include curve interpolation, graph search, sampling-based methods, and numerical optimization algorithms. The patent (CN202410216710.1) discloses a personalized lane change trajectory planning method based on quintic polynomials, which improves the personalization degree of lane change trajectory from "segmented" to "continuous". The proposed personalized lane change trajectory planning method is suitable for structured road lane change scenarios for autonomous driving or assisted driving, and the planned trajectory meets the stability and collision avoidance requirements. Reference (Research on A* algorithm for static path planning based on neighborhood expansion. Guo Xiaojing & Yang Zhuocheng. Computer Engineering and Applications, 2022, 58(8): 168-174. DOI: 10.3778 / j.issn.1002-8331.2010-0222.) In order to solve the problems of low search freedom, long planned path length and large turning angle of the traditional A* algorithm, an improved A* algorithm is proposed. The improved algorithm expands the traditional 8-neighborhood search to 24 neighborhoods, and uses the guidance vector to optimize the number of neighborhoods to improve the search efficiency; the path smoothing algorithm is used to eliminate redundant nodes in the path and optimize the smooth path.

[0004] Although traditional trajectory planning algorithms have great advantages, they generally have the defects of large computational complexity and extreme dependence on environmental information, and poor real-time performance. The rise and rapid development of the field of artificial intelligence has led to the emergence of various bionic intelligent optimization algorithms in the field of trajectory planning, such as ant colony algorithms, genetic algorithms, neural network algorithms, and particle swarm algorithms. The patent (CN202011340491.6) discloses a safe automatic driving lane change trajectory planning method based on an improved LSTM neural network. It uses a trajectory algorithm that considers safety to generate a safe trajectory, and then uses an improved LSTM neural network to learn the safety trajectory and the original trajectory, so that the final lane change trajectory decided is closer to the human riding experience under the premise of safety.

[0005] Intelligent optimization algorithms have good adaptive capabilities and improve real-time capabilities compared to traditional planning algorithms, but they cannot converge quickly, and their ability to adapt to complex environments needs to be improved. They cannot guarantee stability and optimal solutions, so research on path planning based on reinforcement learning algorithms has gradually become more intense. The literature (Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning. KB Naveed, Z. Qiao and JM Dolan. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA, 2021, pp. 601-606) proposed a hierarchical reinforcement learning structure combined with a proportional-integral-derivative (PID) controller for trajectory planning. The results show that the framework reduces the convergence time and helps the car learn in a more dynamic environment.

[0006] Patent (CN202310347395.1) A lane-changing method for autonomous vehicles based on hierarchical deep reinforcement learning, which uses a deep reinforcement learning DQN network and a DDPG network for lane-changing decisions and motion trajectory planning and training. At present, the reinforcement learning-based method can adapt to the ever-changing traffic environment, but the stability is low, and the accuracy of trajectory planning needs to be further improved. Summary of the invention

[0007] In order to improve the stability of an autonomous driving vehicle during lane changing and improve the accuracy of trajectory planning during lane changing, the present invention proposes a lane changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN. This method can realize lane changing decision and trajectory planning for autonomous driving in a complex environment, thereby improving the lane changing efficiency and stability of autonomous driving.

[0008] The technical solutions adopted are:

[0009] A lane-changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN, characterized in that: the specific method is as follows:

[0010] S1. Construct a lane-changing trajectory planning model for an autonomous driving vehicle, wherein the lane-changing trajectory planning model for an autonomous driving vehicle comprises an input layer, a lane-changing layer, and an output layer, wherein the lane-changing layer is composed of a temporal convolutional network TCN and two dual-depth Q networks DDQN, wherein:

[0011] TCN and one DDQN constitute a lane-changing decision layer, and TCN and another DDQN constitute a trajectory planning layer. The environment state space information input through the input layer passes through TCN and is input into the two DDQNs respectively. The lane-changing decision target output by the DDQN in the lane-changing decision layer is input into the DDQN in the trajectory planning layer. The predicted trajectory planning action output by the DDQN in the trajectory planning layer is output via the output layer.

[0012] S2. Obtain historical environment space state information, use the historical environment space state information as a training set, and train the automatic driving lane change trajectory model;

[0013] S3. Use the trained lane change trajectory planning model of the autonomous driving vehicle to predict the autonomous driving lane trajectory.

[0014] In a further preferred embodiment of the technical solution of the present invention, the environment state space information includes at least the coordinates, speed and lane ID of the target lane-changing vehicle and the preceding vehicle in the current lane, the preceding vehicle in the target lane and the following vehicle in the target lane.

[0015] Further preferred embodiment of the technical solution of the present invention, TCN includes a cascaded input layer, N residual blocks, a fully connected layer and an output layer, each residual block includes a 1x1 convolution block and a first dilated causal convolution layer, a first weight normalization layer, a first ReLU activation function, a first Dropout layer, a second dilated causal convolution layer, a second weight normalization layer, a second ReLU activation function, and a second Dropout layer connected in sequence, and the output of the nth residual block is respectively input into the 1x1 convolution block in the n+1th residual block to adjust the number of channels and the first dilated causal convolution layer, and the output of the 1x1 convolution block in the n+1th residual block is superimposed with the output of the second Dropout layer as the input of the n+2th residual block.

[0016] In a further optimization of the technical solution of the present invention, the reward R of the DDQN in the lane change decision layer is g Rewarded by safety R safe and efficiency reward R eff2 Composition, characterized by:

[0017] R g =R safe +R eff2 ;

[0018] The reward R of the trajectory planning layer a Rewarded by safety R safe , efficiency reward R eff and the smoothness reward R smooth Composition, characterized by:

[0019] R a =R safe +Reff +R smooth ;

[0020] Safety Reward R safe Including collision rewards and penalties safe1 and unsafe rewards and penalties R safe2 , characterized by:

[0021] R safe =R safe1 +R safe2 ;

[0022] When the target lane-changing vehicle is in the current lane:

[0023]

[0024] When the target lane-changing vehicle is in the target lane:

[0025]

[0026] Where D is the length of the vehicle, σ1 is the reward constant for collision reward and punishment, Δx CPV is the distance between the target lane-changing vehicle and the vehicle in front of the current lane, Δx TFV is the distance between the target lane-changing vehicle and the vehicle in front of the target lane, Δx TPV is the distance between the target lane-changing vehicle and the vehicle behind in the target lane;

[0027]

[0028] In the formula, Δx r is the expected safe distance between the target lane-changing vehicle and the vehicle in front of the current lane, w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following stage and the lane-changing stage, σ2 is the reward constant for the unsafe rewards and penalties, Carfollowing represents the following stage, and Lane changing represents the lane-changing stage;

[0029] Efficiency Reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​Limit Rewards and Penalties eff2 and the reward and penalty R for not reaching the lane change target range eff3 , characterized by:

[0030] R eff =R eff1 +R eff2 +R eff3

[0031]

[0032] Where w3 is the weight parameter of lane change overtime reward and punishment, σ3 is the reward constant of lane change overtime reward and punishment, Δt is the actual lane change time, and Δtr is the expected lane change time;

[0033]

[0034] In the formula, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and penalty and the minimum speed limit reward and penalty respectively, σ4 is the reward constant of the speed limit reward and penalty, v SV is the target lane-changing vehicle speed;

[0035]

[0036] In the formula, x r1 and x r2 Set the upper and lower limits of the expected range for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range;

[0037] Smoothness reward R smooth Characterized by:

[0038] R smooth =-w6|a t+1 -a t |-w7|k t+1 -k t |

[0039] Where w6 and w7 are the weight parameters corresponding to acceleration bonus and curvature bonus, respectively. t+1 and a t are the vehicle accelerations at time t+1 and t, respectively, k t+1 and k t are the curvatures of the lane changing trajectory at time t+1 and time t respectively.

[0040] In a further preferred embodiment of the technical solution of the present invention, during the training of the automatic driving lane change trajectory model:

[0041] The experience replay pool of the lane change decision layer stores information including the state s at time t t , the historical environment state space information h at time t t , the lane change decision target g at time t t , the reward R of DDQN in the lane change decision layer at time t gt , the state s at time t+1 t+1 And the historical environment state space information h at time t+1 t+1 ;

[0042] The experience replay pool of the trajectory planning layer stores information including the state s at time t t, the historical environment state space information h at time t t , the trajectory planning action a at time t t , the reward R of DDQN in the trajectory planning layer at time t at , the state s at time t+1 t+1 , the historical environment state space information h at time t+1 t+1 And whether the lane change action is completed t .

[0043] In a further preferred embodiment of the technical solution of the present invention, during the training of the automatic driving lane change trajectory model:

[0044] For the lane-changing decision layer, the value function of the target network is:

[0045]

[0046] The loss function is:

[0047]

[0048] The current network parameters are updated as follows:

[0049]

[0050] In the formula, E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer at time t, γ∈[0,1] is the discount factor, α is the learning rate, and R gt is the reward of DDQN in the lane-changing decision layer at time t, g is the set of all possible lane-changing vehicles, s t is the state at time t, s t+1 is the state at time t+1;

[0051] For the trajectory planning layer, the value function of the target network is:

[0052]

[0053] The loss function is:

[0054]

[0055] The current network parameters are updated as follows:

[0056]

[0057] In the formula, R at is the reward of DDQN in the trajectory planning layer at time t, a is the set of all possible trajectory planning actions, and are the current network parameters and target network parameters of the trajectory planning layer at time t respectively.

[0058] In a further preferred embodiment of the technical solution of the present invention, during the training process of the automatic driving lane changing trajectory model: a greedy strategy is used to select the lane changing decision target and trajectory planning action.

[0059] In a further optimization of the technical solution of the present invention, during the training process of the automatic driving lane changing trajectory model: taking improving the lane changing success rate as the optimization goal, multiple rounds of training are performed to obtain the automatic driving lane changing trajectory model parameters with the optimal lane changing success rate.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The lane-changing trajectory planning method for an autonomous driving vehicle based on a hierarchical DDQN and TCN hybrid model of the present invention can adapt to the ever-changing complex traffic environment, has good stability, can cope with complex control problems, and provides technical support for the autonomous driving lane-changing system.

[0062] 2. The lane changing decision model in the lane changing decision layer and the trajectory planning model in the trajectory planning layer of the present invention both use TCN to process data, which helps DDQN to better understand the relationship and dynamic changes between vehicles, thereby making more accurate target decisions and trajectory planning actions, and helps to improve the lane changing efficiency and stability of autonomous driving.

[0063] 3. The present invention improves the reward function, thereby ensuring the stability of vehicle lane change while improving the comfort of passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is an autonomous driving vehicle lane change trajectory planning model architecture based on a hybrid model of layered dual-depth Q-network and temporal convolutional network;

[0065] Figure 2(a) is a schematic diagram of causal expansion convolution;

[0066] Figure 2 (b) TCN residual block;

[0067] Figure 2 (c) TCN data processing steps;

[0068] Figure 3 It is a training process for the lane-changing trajectory planning method of an autonomous driving vehicle based on a hybrid model of a layered dual-depth Q network and a temporal convolutional network;

[0069] Figure 4 This is a schematic diagram of the lane changing process. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following Figure 1-4, the present invention is described in further detail.

[0071] It should be understood that the description herein is only used to explain the present invention, and is not used to limit the present invention.

[0072] The technical solution adopted by the present invention is a lane-changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of a layered dual-depth Q network and a temporal convolutional network, comprising the following two steps:

[0073] Step A. Construction of lane-changing trajectory planning model for autonomous driving vehicles based on HDDQN+TCN hybrid model;

[0074] Step B. Training and testing of the lane-changing trajectory planning model for the autonomous vehicle.

[0075] The lane-changing trajectory planning model for the autonomous driving vehicle constructed in step A includes an input layer, a lane-changing layer, and an output layer, such as Figure 1 shown.

[0076] A.1 Input layer: The input of the lane change trajectory planning model of the autonomous driving vehicle is the historical environment state space information h t , consisting of the states s at multiple moments t Composition, that is, h t ={s1,s2…s t The state s specifically includes the coordinates, speed, lane ID and other information of the target lane-changing vehicle and surrounding vehicles, which can be specifically represented by the following formula: s = [lane IDSV ,lane IDCPV ,lane IDTPV ,lane IDTFV ,v SV ,v CPV ,v TPV ,v TFV ,Δx CPV ,Δx TPV ,Δx TFV

[0077] In the formula, lane IDSV The lane ID of the target lane-changing vehicle;

[0078] Lane IDCPV 、lane IDTPV and lane IDTFV They are the lane IDs of the front vehicle in the current lane, the front vehicle in the target lane, and the rear vehicle in the target lane;

[0079] v SV is the target lane-changing vehicle speed;

[0080] v CPV 、v TPV and v TFVare the speeds of the front vehicle in the current lane, the front vehicle in the target lane, and the rear vehicle in the target lane respectively;

[0081] Δx CPV =|x CPV -x SV | is the distance between the target lane-changing vehicle and the vehicle in front of the current lane, x SV and x CPV are the position coordinates of the center of the target lane-changing vehicle and the center of the front vehicle in the current lane along the lane direction respectively;

[0082] Δx TPV =|x TPV -x SV | is the distance between the target lane-changing vehicle and the vehicle in front of the target lane, x TPV is the position coordinate of the center of the front vehicle in the target lane along the lane direction;

[0083] Δx TFV =|x SV -x TFV |Distance between the target lane-changing vehicle and the vehicle behind in the target lane, x TFV is the position coordinate of the center of the vehicle behind the target lane along the lane direction.

[0084] A.2 Lane-changing layering: The lane-changing layering structure includes a lane-changing decision layer and a trajectory planning layer. Each layer adopts a DDQN and TCN architecture. The lane-changing decision layer uses the historical environment state space information and the reward R g The lane-changing decision model is trained to fit the value function Q, and the current network is used to select a target to maximize the current network value. The target lane-changing vehicle selects the lane-changing decision target with a probability of (1-ε) according to the greedy strategy.

[0085] Among them, the trajectory planning layer uses the historical environment state space information, lane change decision goals and reward R a The trajectory planning model is trained and fitted, and the current network is used to select a trajectory planning action to maximize the current network value. The target lane-changing vehicle selects the lane-changing trajectory planning action according to the greedy strategy with a probability of (1-ε).

[0086] As for the TCN part in the layer, it is the core feature extractor of the entire lane change layer, capturing the long-term dependency of the historical environment state space information in the input layer, so as to process the data containing noise or inaccurate data, improve the generalization ability of the model, and improve the accuracy of subsequent lane change decisions and trajectory planning. This is achieved through the following steps, as shown in Figure 2(c):

[0087] First, the input of TCN is the historical environment state space information h t .

[0088] Secondly, the data filling method in the TCN data processing process is set to causal dilation convolution. Specifically, the neuron output of the TCN at time t is only related to s1, s2…s t Related to the future state t+1 ,s t+2 ,s t+3 …s t+n And by sampling the TCN input data layer by layer at an exponential level, a larger receptive field can be obtained with fewer network layers, as shown in Figure 2(a). The specific calculation method is represented by the following formula:

[0089]

[0090] Where F(s) is the convolution operation on the sequence element s,

[0091] f is the number of filters,

[0092] k is the size of the convolution kernel,

[0093] d is the expansion factor,

[0094] x s-d·i To perform convolution only on the past state,

[0095] * is the convolution operation.

[0096] At this point, the receptive field can be characterized as:

[0097] field=(k-1)·d+1

[0098] Furthermore, the network depth is increased, and residual blocks are used instead of convolutional layers to suppress the gradient vanishing or gradient exploding problems caused by the increase in network depth. Residual links are used to transfer information across layers in the network structure. The structure of the residual block is shown in Figure 2(b). It has two layers of dilated causal convolution and ReLu activation function. Weight normalization layers and Dropout are added to each layer to regularize the network and improve the generalization ability of the network.

[0099] Furthermore, the TCN network parameters are set, including the input layer shape, output layer dimension, number of filters, convolution kernel size and other parameters, as shown in Table 1 below:

[0100] Table 1 TCN network parameters

[0101]

[0102] Finally, the output of TCN is an abstract expression of the characteristics and temporal relationship of the input information, capturing the evolution and changes of the input information over time and understanding the temporal pattern and structure in the input information. It will be used as the input of the lane change decision layer and trajectory planning layer.

[0103] For the reward R in the DDQN structure, there are three types: safety reward, efficiency reward, and smoothness reward.

[0104] Among them, the reward R of the lane change decision layer g Rewarded by safety R safe and efficiency rewards (only speed limit rewards and penalties) R eff2 Composition, represented by the following formula:

[0105] R g =R safe +R eff2

[0106] Among them, the reward R of the trajectory planning layer a Rewarded by safety R safe , efficiency reward R eff and the smoothness reward R smooth Composition, represented by the following formula:

[0107] R a =R safe +R eff +R smooth

[0108] The three types of rewards are obtained in the following ways:

[0109] (1) Safety reward R safe Including collision rewards and penalties safe1 and unsafe rewards and penalties R safe2 :

[0110] R safe =R safe1 +R safe2

[0111] Among them, the collision reward and penalty uses the center point of the vehicle as the coordinate point. During the following or lane changing stage, when the distance between the target lane changing vehicle and the preceding or following vehicle in the same lane is less than or equal to the vehicle length D, it is considered a collision and a corresponding penalty is given. Otherwise, a corresponding reward is given, which is represented by the following formula:

[0112] When the target lane-changing vehicle is in the current lane:

[0113]

[0114] When the target lane-changing vehicle is in the target lane:

[0115]

[0116] Where D is the length of the vehicle (m), and σ1 is the reward constant for collision reward and punishment.

[0117] Among them, unsafe rewards and penalties, the target lane-changing vehicle needs to maintain a certain expected safe distance from the vehicle in front of the current lane during the following or lane-changing process to ensure the safety of the vehicle and respond to emergencies, which is represented by the following formula:

[0118]

[0119] In the formula, Δx r is the expected safe distance between the target lane-changing vehicle and the vehicle in front of the current lane, w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following stage and the lane-changing stage, respectively, and σ2 is the reward constant for the unsafe rewards and penalties.

[0120] (2) Efficiency Reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​Limit Rewards and Penalties eff2 and the reward and penalty R for not reaching the lane change target range eff3 , represented by the following formula:

[0121] R eff =R eff1 +R eff2 +R eff3

[0122] Among them, the lane change timeout reward and penalty, when the lane change decision layer outputs the option of lane change, the expected lane change time is set for the target lane change vehicle to ensure the lane change efficiency, which is represented by the following formula:

[0123]

[0124] Where w3 is the weight parameter of lane change overtime reward and punishment, σ3 is the reward constant of lane change overtime reward and punishment, Δt is the actual lane change time, and Δt r is the expected lane change time.

[0125] Among them, the speed limit reward and punishment, when the target lane-changing vehicle is driving, it is necessary to set a speed limit for the target lane-changing vehicle to improve the traffic efficiency of the road while ensuring safety, which is represented by the following formula:

[0126]

[0127] In the formula, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and punishment and the minimum speed limit reward and punishment respectively, and σ4 is the reward constant of the speed limit reward and punishment.

[0128] Among them, the reward and punishment for not reaching the lane changing target range, when the lane changing decision layer outputs the option of lane changing, the expected range is set for the final lane changing point of the target lane changing vehicle to ensure that the vehicle selects the appropriate lane changing trajectory and improves the lane changing efficiency, which is represented by the following formula:

[0129]

[0130] In the formula, x r1 and x r2 The upper and lower limits of the expected range are set for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range.

[0131] (3) Smoothness reward R smooth In addition to ensuring safety and efficiency, the autonomous driving vehicle also needs to provide the driver and passengers with a good riding experience. The acceleration a and the curvature k of the lane change trajectory are used as constraints to ensure the comfort of the vehicle during driving, which is represented by the following formula:

[0132] R smooth =-w6|a t+1 -a t |-w7|k t+1 -k t |

[0133] Where w6 and w7 are the weight parameters corresponding to acceleration reward and penalty and curvature reward and penalty respectively.

[0134] A.3 Output layer: In the lane change layer, the lane change decision layer can output the lane change decision target, including lane changing or following; the trajectory planning layer can output the predicted trajectory planning action, including the trajectory coordinates and the speed of the corresponding coordinates.

[0135] Among them, step B includes three parts: autonomous driving vehicle lane change trajectory planning model training process, training data set generation, and model optimization.

[0136] B. In the lane-changing trajectory planning model of an autonomous vehicle, the parameters include historical environment state space information, lane-changing decision target set, trajectory planning action set, rewards, etc. The adjustable parameters include discount factor, exploration rate, number of TCN convolution layers, convolution kernel size, number of filters, expansion coefficient, etc.

[0137] First, the networks of the lane change decision layer and trajectory planning layer are initialized with random current network parameters θ g and θ a Initialize the current network and copy it to the target network. The lane change decision layer and trajectory planning layer each have an experience replay pool D1 and D2. Initialize the experience replay pools D1 and D2, set the maximum capacity of the replay memory, set the number of training times, and set the time limit for each round.

[0138] Each round of training consists of several episodes, and each episode consists of several time steps.

[0139] In each round, the historical environment state space information is initialized, the TCN is used to process the historical environment state space information, and the relevant variables are initialized. The relevant variables include collision history records, rewards, and other related information generated during the execution of the action.

[0140] In each time step, the target lane-changing vehicle selects the lane-changing decision target and trajectory planning action based on the historical environment state space information.

[0141] For the lane-changing decision layer, based on the processed historical environment state space information, Q(s, g, θ g ) to approximate the value function Q of the optimal decision target * (s,g). Then use the current network and use the state s t+1 Select a goal to maximize the current network value. And the target lane-changing vehicle selects the optimal lane-changing decision target with a probability of (1-ε) according to the greedy strategy. It can be specifically characterized as:

[0142]

[0143] In the formula, g is the set of all possible targets, The current network is in state s t+1 The objective value function under .

[0144] For the trajectory planning layer, based on the target output by the lane change decision layer and the processed historical environment state space information, Q(s, a; g, θ a ) to approximate the value function Q of the optimal trajectory planning action * (s,a;g). Then use the current network and use the state s t+1 Select an action to maximize the current network value. And use the greedy strategy to select the optimal trajectory planning action with a probability of (1-ε). It can be specifically characterized as:

[0145]

[0146] In the formula, a is the set of all possible actions, The current network is in state s t+1 The action-value function under .

[0147] Execute the action and get the new state s t+1 , the reward R of the lane-changing decision layer g , the reward R of the trajectory planning layer a And other information.

[0148] Store the current state s in the experience replay pool D1 t , historical environment state space information h t , Lane change decision target g t , Lane-changing decision-making layer reward R gt , new state s t+1 , new historical environment state space information h t+1 etc.; store the current state s in the experience replay pool D2 t , historical environment state space information h t , trajectory planning action a t , trajectory planning layer reward R at , new state s t+1 , new historical environment state space information h t+1 And whether the action is completed.

[0149] In each time step, a small batch of samples D is randomly drawn from the experience replay pool to calculate the lane change decision layer value function And the trajectory planning layer value function The target value is compared with the current network value function to calculate the loss function of the lane change decision layer and the trajectory planning layer respectively. and Update the current network parameters using the gradient descent method and

[0150] For the lane-changing decision layer, the value function of the target network is:

[0151]

[0152] The loss function is obtained as:

[0153]

[0154] Then the current network parameters are updated as follows:

[0155]

[0156] In the formula, E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer, respectively. The former is used to select the target (the strategy of the current state), and the latter is used to evaluate the value of the current target. γ∈[0,1] is the discount factor, and α is the learning rate.

[0157] For the trajectory planning layer, the value function of the target network is:

[0158]

[0159] The loss function is obtained as:

[0160]

[0161] Then the current network parameters are updated as follows:

[0162]

[0163] In the formula, and They are the current network parameters and target network parameters of the trajectory planning layer. The former is used to select actions (the strategy of the current state), and the latter is used to evaluate the value of the current action.

[0164] Update the target network parameters θ every several time steps g′ =θ g ,θ a′ =θ a .

[0165] The above process is repeated, and the round ends when the vehicle collides, reaches the set time limit, or the autonomous vehicle completes the lane change.

[0166] Repeat the above steps and continue training until all rounds in a round of training are completed, and the first round of training is over.

[0167] With the improvement of lane-changing success rate as the optimization goal, multiple rounds of training are carried out, and the lane-changing model parameters are continuously adjusted according to the training results to determine the model parameters that can optimize the lane-changing success rate.

[0168] B.2 Training data set generation: mainly the real-time motion information of the target lane-changing vehicle and surrounding vehicles during the lane-changing process, including coordinates, vehicle lane ID, speed, etc. The lane-changing process in the real-time motion information is calibrated by the change in the horizontal coordinates of the target lane-changing vehicle.

[0169] The lane-changing process is divided into the lane-changing decision phase (e.g. Figure 4 AB phase), lane change execution phase (such as Figure 4 BC phase) and lane change completion phase (such as Figure 4 Among them, the historical environment state space information is used as input, the lane change decision stage corresponds to the lane change decision layer in step A, and the lane change decision target is output. The lane change execution stage and the lane change completion stage correspond to the trajectory planning layer in step A, and the trajectory planning action is output.

[0170] B.3 Optimization of the lane-changing trajectory planning model for autonomous vehicles: Use the training data set to train the model. According to the training results, such as the loss function value and the lane-changing success rate, adjust the model parameters, such as the number of TCN convolution layers, convolution kernel size, number of filters, expansion coefficient, discount factor, and exploration rate. Repeat the model training under different parameters to determine the optimal parameters for the model training results.

[0171] The present invention proposes an improved hierarchical lane-changing trajectory planning model based on the temporal convolutional network (TCN). A dual deep Q network is used for training and testing of the lane-changing decision layer and the trajectory planning layer. Each layer uses TCN to process data, which helps the DDQN model to better understand the relationship and dynamic changes between vehicles, thereby making more accurate decisions and planning, and helps to improve the lane-changing efficiency and stability of autonomous driving.

[0172] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

[0173] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A lane-changing trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN, characterized by: The specific method is as follows: S1. Construct a lane-changing trajectory planning model for an autonomous driving vehicle, wherein the lane-changing trajectory planning model for an autonomous driving vehicle comprises an input layer, a lane-changing layer, and an output layer, wherein the lane-changing layer is composed of a temporal convolutional network TCN and two dual-depth Q networks DDQN, wherein: TCN and one DDQN constitute a lane-changing decision layer, and TCN and another DDQN constitute a trajectory planning layer. The environment state space information input through the input layer passes through TCN and is input into the two DDQNs respectively. The lane-changing decision target output by the DDQN in the lane-changing decision layer is input into the DDQN in the trajectory planning layer. The predicted trajectory planning action output by the DDQN in the trajectory planning layer is output via the output layer. S2. Obtain historical environment space state information, use the historical environment space state information as a training set, and train the automatic driving lane change trajectory model; S3. Use the trained lane change trajectory planning model of the autonomous driving vehicle to predict the autonomous driving lane trajectory.

2. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: The environment state space information includes at least the coordinates, speed and lane ID of the target lane-changing vehicle and the preceding vehicle in the current lane, the preceding vehicle in the target lane and the following vehicle in the target lane.

3. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: TCN includes a cascaded input layer, N residual blocks, a fully connected layer and an output layer. Each residual block includes a 1x1 convolution block and a first dilated causal convolution layer, a first weight normalization layer, a first ReLU activation function, a first Dropout layer, a second dilated causal convolution layer, a second weight normalization layer, a second ReLU activation function, and a second Dropout layer connected in sequence. The output of the nth residual block is respectively input into the 1x1 convolution block in the n+1th residual block to adjust the number of channels and the first dilated causal convolution layer. The output of the 1x1 convolution block in the n+1th residual block is superimposed with the output of the second Dropout layer as the input of the n+2th residual block.

4. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: The reward R of DDQN in the lane-changing decision layer g Rewarded by safety R safe and efficiency reward R eff2 Composition, characterized by: R g =R safe +R eff2 ; The reward R of the trajectory planning layer a Rewarded by safety R safe , efficiency reward R eff and the smoothness reward E smooth Composition, characterized by: R a =R safe +R eff +R smooth ; Safety Reward R safe Including collision reward R safe1 and unsafe rewards and penalties R safe2 , characterized by: R safe =R safe1 +R safe2 ; When the target lane-changing vehicle is in the current lane: When the target lane-changing vehicle is in the target lane: Where D is the length of the vehicle, σ1 is the reward constant for collision reward and punishment, Δx CPV is the distance between the target lane-changing vehicle and the vehicle in front of the current lane, Δx TFV is the distance between the target lane-changing vehicle and the vehicle in front of the target lane, Δx TPV is the distance between the target lane-changing vehicle and the vehicle behind in the target lane; In the formula, Δx r is the expected safe distance between the target lane-changing vehicle and the vehicle in front of the current lane, w1 and w2 are the weight parameters corresponding to the unsafe rewards and penalties in the following stage and the lane-changing stage, σ2 is the reward constant for the unsafe rewards and penalties, Carfollowing represents the following stage, and Lane changing represents the lane-changing stage; Efficiency Reward R eff Including lane change timeout reward and penalty R eff1 , Speed ​​Limit Rewards and Penalties eff2 and the reward and penalty R for not reaching the lane change target range eff3 , characterized by: R eff =R eff1 +R eff2 +R eff3 Where w3 is the weight parameter of lane change overtime reward and punishment, σ3 is the reward constant of lane change overtime reward and punishment, Δt is the actual lane change time, and Δt r is the expected lane change time; In the formula, v r1 is the maximum speed limit of the vehicle during driving, v r2 is the minimum speed limit during vehicle driving, w4 and w5 are the weight parameters corresponding to the maximum speed limit reward and penalty and the minimum speed limit reward and penalty respectively, σ4 is the reward constant of the speed limit reward and penalty, v SV is the target lane-changing vehicle speed; In the formula, x r1 and x r2 Set the upper and lower limits of the expected range for the final lane-changing point of the target lane-changing vehicle, and σ5 is the reward constant for rewards and penalties for failing to reach the lane-changing target range; Smoothness reward R smooth Characterized by: <h2 style=";text-align:left;direction:ltr">R<h2 style=";text-align:left;direction:ltr"> smooth <h2 style=";text-align:left;direction:ltr"> =-w6|a<h2 style=";text-align:left;direction:ltr"> t+1 <h2 style=";text-align:left;direction:ltr"> -a<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> |-w7|k<h2 style=";text-align:left;direction:ltr"> t+1 <h2 style=";text-align:left;direction:ltr"> -k<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> | Where w6 and w7 are the weight parameters corresponding to acceleration bonus and curvature bonus, respectively. +1 and a t are the vehicle acceleration at time t+1 and time t, k t+1 and k t are the curvatures of the lane changing trajectory at time t+1 and time t respectively.

5. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: The experience replay pool of the lane change decision layer stores information including the state s at time t t , the historical environment state space information h at time t t , the lane change decision target g at time t t , the reward R of DDQN in the lane change decision layer at time t gt , the state s at time t+1 t+1 And the historical environment state space information h at time t+1 t+1 ; The experience replay pool of the trajectory planning layer stores information including the state s at time t t , the historical environment state space information h at time t r , the trajectory planning action a at time t t , the reward R of DDQN in the trajectory planning layer at time t at , the state s at time t+1 t+1 , the historical environment state space information h at time t+1 t+1 And whether the lane change action is completed t .

6. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: For the lane-changing decision layer, the value function of the target network is: The loss function is: The current network parameters are updated as follows: In the formula, E[·] represents expectation, and are the current network parameters and target network parameters of the lane-changing decision layer at time t, γ∈[0,1] is the discount factor, α is the learning rate, and R gt is the reward of DDQN in the lane-changing decision layer at time t, g is the set of all possible lane-changing vehicles, s t is the state at time t, s t+1 is the state at time t+1; For the trajectory planning layer, the value function of the target network is: The loss function is: The current network parameters are updated as follows: In the formula, R at is the reward of DDQN in the trajectory planning layer at time t, a is the set of all possible trajectory planning actions, and are the current network parameters and target network parameters of the trajectory planning layer at time t respectively.

7. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: A greedy strategy is used to select lane-changing decision targets and trajectory planning actions.

8. The lane change trajectory planning method for an autonomous driving vehicle based on a hybrid model of HDDQN and TCN according to claim 1, characterized in that: During the training process of the autonomous driving lane change trajectory model: With the optimization goal of improving the lane change success rate, multiple rounds of training are performed to obtain the parameters of the autonomous driving lane change trajectory model with the optimal lane change success rate.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 8 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images