Vehicle track optimization control method for automatic driving lane
By constructing the DDPG follow-up model and DQN lane change model on the autonomous driving dedicated lane, combined with the deep reinforcement learning algorithm, the vehicle trajectory control is optimized, and the problem of insufficient impact of dedicated lanes on the lane change selection in the existing technology is solved, and the problem of insufficient impact of dedicated lanes on the lane change selection in autonomous driving vehicles is improved. Passage efficiency and safety.
Patent Information
- Application Number
- CN202410528158.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-29
- Publication Date
- 2024-12-06
AI Technical Summary
The existing vehicle trajectory optimization control methods mostly focus on the target vehicle and surrounding vehicles, and do not fully consider the impact of dedicated lanes on the lane selection of autonomous driving vehicles, resulting in insufficient traffic efficiency and safety under mixed traffic flow conditions.
A vehicle trajectory optimization control method for autonomous driving special lanes was designed. By constructing a DDPG-based follow-up model and DQN lane change model, combined with deep reinforcement learning algorithms, training the follow-up and lane change decisions of the agent, considering the vehicle Coordinated optimization with road sections to optimize vehicle trajectory control.
Effectively improve the behavior control of following and lane change on dedicated lanes by autonomous driving vehicles, improve traffic efficiency and safety, reduce congestion, promote more autonomous driving vehicles to switch lanes to dedicated lanes, and improve overall road access efficiency.
Smart Images

Figure CN119078871A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent transportation and traffic control, and specifically relates to a vehicle trajectory optimization control method for an automatic driving dedicated lane. Background Art
[0002] With the continuous development of my country's economic construction, the car ownership rate of residents has continued to increase, and the growing travel demand has brought tremendous pressure on road safety and energy issues. Therefore, improving the efficiency of the traffic system has become a top priority. The emergence of autonomous driving technology provides a new solution to these problems. Compared with manual driving, autonomous driving vehicles have obvious advantages in reducing traffic accidents, relieving traffic pressure, and reducing driving costs and time costs. Autonomous driving technology makes following control and lane change control possible by relying on the ability of on-board equipment to exchange information with surrounding vehicles in the Internet of Vehicles. In the following state, the vehicle can be kept at a specific speed or at a certain distance from the vehicle in front. In the lane change process, the target vehicle can be made to change lanes in a certain mode. After obtaining information such as the type, speed, headway, etc. of the surrounding vehicles, the vehicle's following acceleration and lane change intention are controlled through a predetermined vehicle control strategy, thereby reducing speed fluctuations in the queue, improving traffic efficiency, and improving lane change safety. Deep reinforcement learning is a behavioral decision-making method that does not rely on artificial rules. It obtains decision-making actions that increase reward values through the interaction between the intelligent agent and the environment, and plays a huge role in solving complex problems in the intelligent driving environment. At present, vehicle trajectory optimization control methods mostly focus on the target vehicle and surrounding vehicles, and have a low degree of integration with lane function design. In fact, dedicated lanes have a certain impact on the lane change choice of autonomous driving vehicles. Summary of the invention
[0003] The purpose of the present invention is to propose a vehicle trajectory optimization control method for an autonomous driving dedicated lane. The dedicated lane application scenario is a single dedicated lane on the left side of a three-lane system. The control methods are a following model based on DDPG to evaluate and output the following action of the intelligent agent and a lane changing model based on DQN to evaluate and output the lane changing intention of the intelligent agent.
[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a vehicle trajectory optimization control method for an autonomous driving dedicated lane, comprising the following steps:
[0005] S1. Design application scenarios of dedicated lanes under mixed traffic flow conditions;
[0006] S2. Design the Markov reward process of the car-following model, determine the state space, action space and reward value, build a network framework based on the deep deterministic policy gradient algorithm (DDPG), and train the car-following decision of the intelligent agent;
[0007] S3. Design the Markov reward process of the lane-changing model, determine the state space, action space and reward value, build a network framework based on the deep reinforcement learning algorithm (DQN), and train the lane-changing decision of the intelligent agent;
[0008] S4. Under the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road to achieve vehicle trajectory optimization control.
[0009] Furthermore, in the aforementioned step S1, the application scenarios of dedicated lanes under mixed traffic flow conditions include: a multi-lane traffic environment, which includes autonomous driving dedicated lanes and non-autonomous driving dedicated lanes.
[0010] Furthermore, in the aforementioned step S2, a Markov reward process is constructed for a specific autonomous driving method, including the following steps:
[0011] S21, construct state space s(t) = {V f (t),a f (t),D f (t),V r (t),T w (t), Delay(t)}, where V f (t) is the current following vehicle speed, a f (t) is the current following vehicle acceleration, D f (t) is the current following distance, V r (t) is the relative speed to the preceding vehicle, T w (t) is the average waiting time of vehicles on the road section, and Delay(t) is the average delay time of vehicles on the road section;
[0012] S22, construct action space a(t)∈[-3,3](m / s 2 ),
[0013] The following safety distance constraint is
[0014] S23. Design the reward function. The specific steps are as follows:
[0015] The total reward value is
[0016] R cf =w safety ×R safety +w effi ×R effi +w comf ×R comf +w fuel ×R fuel
[0017] Where R safetyis the security bonus, R effi is the efficiency reward, R comf For comfort bonus, R fuel For fuel consumption reward, w safety ,w effi ,w comf w fuel They are the weights of the corresponding rewards, specifically:
[0018] (1) Security Rewards
[0019] R safety =D f -D s
[0020] D f The intelligent agent calculates the safe distance that will not cause a collision within the driver's reaction time based on its own following speed and the speed of the preceding vehicle. s The following distance maintained by the current agent;
[0021] (2) Efficiency Rewards
[0022] R effi =A×R speed +B×R totalwait +C×R totaldelay
[0023] Where A, B, and C are the weights of each reward, which can be further divided into three factors: (a) the deviation of the following vehicle speed before and after the action; (b) the numerical difference of the total waiting time of the road section before and after the action; (c) the numerical difference of the total vehicle delay before and after the action;
[0024] a) Deviation of the following vehicle speed before and after the action
[0025] R speed =V f (t+1)-V f (t)
[0026] V f (t) is the following speed of the agent before taking action, V f (t+1) is the following speed after adopting the following action;
[0027] b) Section waiting time
[0028] R totalwait =TW total (t)-TW total (t+1)
[0029] TW total (t) is the total waiting time before the agent selects an action, TW total(t+1) is the total waiting time of the road section after the agent selects an action;
[0030] c) The difference in the total delay of vehicles on the road section before and after the action
[0031] R totaldelay =Delay(t)-Delay(t+1)
[0032] Delay(t) is the total delay of the road section before the agent adopts the action, and Delay(t+1) is the total delay of the road section after adopting the following action;
[0033] (3) Comfort Reward
[0034]
[0035] jerk and jerk′ are the comfort parameters before and after the action execution, respectively;
[0036] (4) Fuel consumption reward
[0037] R fuel =P′ fuel -P fuel
[0038] P fuel ,P′ fuel They are the fuel consumption before and after the action is executed;
[0039] S24. Build a DDPG-based network framework to train the agent's car-following decision:
[0040] (1) Initialize the Critic evaluation network structure Q(s,a|θ) Q ) and Actor decision network structure μ(s|θ μ ) and its weight θ Q and θ μ , initialize the Critic target - evaluate the network structure Q′(s′,a′|θ Q′ ) and Actor goal-decision network structure μ′(s′|θ μ′ ) and its weight θ Q′ and θ μ′ , and iterate step (2) in each subsequent step;
[0041] (2) Initialize a random process K for action exploration; initialize state s1 as the first state of the current state sequence, obtain its feature network, and iterate steps (3) to (10) in each subsequent step;
[0042] (3) According to the current state s t Strategy and exploration noise K t Select the following acceleration action at =μ(s t |θ μ )+K t ;
[0043] (4) Vehicle performs action a t And observe the reward r t And observe the new state s t+1 ;
[0044] (5) The interaction data of the intelligent autonomous driving vehicle (s t ,a t ,r t ,s t+1 ) is stored in the experience pool;
[0045] (6) Sample random mini-batches of N transformations in the replay experience pool;
[0046] (7) The Critic evaluation network calculates the current Q value y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ );
[0047] (8) Update the critic evaluation network parameters by minimizing the loss. The loss value is expressed as
[0048] (9) Use the sampled policy gradient to update the gradient of the Actor policy network;
[0049] (10) Update the parameters θ of the Critic target-evaluation network and the Actor target-decision network Q′ and θ μ′ .
[0050] Furthermore, in the aforementioned step S3, a Markov reward process is constructed for a specific autonomous driving method:
[0051] S31. Constructing the state space Where a is the vehicle acceleration, v is the vehicle speed, d is the relative distance between the agent and the surrounding vehicles, (l,f,lf,ll,rf,rl) are the front vehicle, the rear vehicle, the left front vehicle, the left rear vehicle, the right front vehicle, and the right rear vehicle, respectively. lon ,x lat is the position of the agent, Lane_index is used to indicate the lane number, and Occ is the occupancy rate of the target road section;
[0052] S32, constructing an action space a(t) = [-1, 0, 1], where -1 indicates changing lanes to the right non-dedicated lane, 0 indicates keeping the current lane, and 1 indicates changing lanes to the left dedicated lane;
[0053] S33. Design a reward function. The reward function and the car-following model have some repeated calculation methods. The specific steps are as follows:
[0054] The total reward value is
[0055] R lc =w safety ×R safety +w effi ×R effi +w comf ×R comf +w fuel ×R fuel +w adv ×R adv
[0056] w safety ,w effi ,w comf ,w fuel ,w adv They are the weights of the corresponding rewards, specifically:
[0057] (1) Security Rewards
[0058]
[0059] Specify penalties and rewards for collisions and non-collisions respectively;
[0060] (2) Efficiency Rewards
[0061] R effi =-|v agent -v desired |
[0062] Among them, v agent is the current speed of the agent, v desired is the expected speed, set to the road speed limit;
[0063] (3) Comfort Reward
[0064]
[0065] The setting method is the same as the corresponding reward setting method of the car-following model;
[0066] (4) Fuel consumption reward
[0067] R fuel =P′ fuel -P fuel
[0068] The setting method is the same as the corresponding reward setting method of the car-following model;
[0069] (5) Lane Change Advantage Reward
[0070] R adv =A×R Lane_index +B×R Occ
[0071] Where A and B are the weights of each reward, which can be further divided into two factors: (a) the reward or punishment for whether the driverless vehicle chooses the dedicated lane; (b) the change in lane occupancy before and after the action;
[0072] a) Rewards and penalties for autonomous vehicles choosing dedicated lanes
[0073]
[0074] Reward or punish driverless vehicles based on whether they choose dedicated lanes;
[0075] b) Lane occupancy rate changes before and after the action
[0076] R Occ =logOcc(t)-logOcc(t+1)
[0077] Occ(t) and Occ(t+1) are the occupancy rates of the road section before and after the agent takes the action;
[0078] S34. Build a DQN-based network framework to train the lane-changing decision of the agent, and select the quintic polynomial for the lane-changing trajectory:
[0079] DQN selects the best action based on the Q table, where a dynamic ε-greedy exploration strategy is selected. The learning probability ε value decreases with the increase in the number of iterations. A random number is selected during iteration. If it is greater than ε, the action with the largest Q value is selected. If it is less than ε, random exploration is performed. This makes the strategy selection biased towards the exploration mode in the early stage of learning and biased towards the utilization mode in the later stage of learning. Among them: ε′ is the subsequent learning probability, ε initial ,ε final are the initial and final ε respectively; considering the situation where the target vehicle cannot change lanes, that is, the target vehicle changes lanes from the left lane to the left or from the right lane to the right, the Q value of the unexecutable strategy in the current state is eliminated to avoid invalid exploration when the maximum Q value is selected.
[0080] Furthermore, the specific implementation steps of the aforementioned step S4 are:
[0081] In the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road. When making a lane changing decision, if the lane changing strategy is executed, the status of the leading vehicle is updated. If the lane changing strategy is not executed, a following decision is made to control the following vehicle acceleration to achieve vehicle trajectory optimization control.
[0082] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:
[0083] 1. The present invention combines the dedicated lane function design under specific scenarios with the vehicle control method, and proposes a vehicle trajectory optimization control method for the dedicated lane for autonomous driving. The vehicle control strategy is divided into a following strategy and a lane-changing strategy. The following model effectively solves the acceleration control problem in the following behavior, and the lane-changing strategy effectively solves the lane-changing intention problem in the lane-changing behavior.
[0084] 2. The present invention takes into account the coordinated optimization of the target vehicle and the target road section in the following behavior decision and the lane changing behavior decision. It not only takes into account the microscopic traffic parameters such as the speed and distance between the target vehicle and the surrounding vehicles, but also takes into account the macroscopic traffic conditions, and adds the average waiting time and average delay time of the target road section. This allows the intelligent agent to improve the global vehicle traffic efficiency while improving its own following efficiency.
[0085] 3. The present invention adds a dedicated lane selection reward that takes occupancy rate into consideration in the lane changing model, which encourages more autonomous driving vehicles to change lanes to dedicated lanes, thereby improving the overall traffic efficiency of the road section and avoiding congestion. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] Figure 1 It is a schematic flow chart of the method of the present invention;
[0087] Figure 2 It is a schematic diagram of the algorithm framework of the car-following model of the present invention;
[0088] Figure 3 Schematic diagram of the lane changing model algorithm framework of the present invention. DETAILED DESCRIPTION
[0089] In order to better understand the technical content of the present invention, a three-lane application scenario is given as an example and explained with the attached drawings.
[0090] like Figure 1 As shown, the present invention provides a vehicle trajectory optimization control method for an autonomous driving dedicated lane, comprising the following steps:
[0091] S1. Design application scenarios of dedicated lanes under mixed traffic flow conditions;
[0092] S2. Design the Markov reward process of the car-following model, determine the state space, action space and reward value, build a network framework based on the deep deterministic policy gradient algorithm (DDPG), and train the car-following decision of the intelligent agent;
[0093] S3. Design the Markov reward process of the lane-changing model, determine the state space, action space and reward value, build a network framework based on the deep reinforcement learning algorithm (DQN), and train the lane-changing decision of the intelligent agent;
[0094] S4. Under the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road to achieve vehicle trajectory optimization control.
[0095] Furthermore, the specific scenario of step S1 is: a three-lane traffic environment, in which the left lane is a dedicated lane for autonomous driving, and the middle and right lanes are dedicated lanes for non-autonomous driving.
[0096] refer to Figure 2 , including the following steps:
[0097] Construct a Markov reward process for a specific autonomous driving method:
[0098] S21, construct state space s(t) = {V f (t),a f (t),D f (t),V r (t),T w (t), Delay(t)}, where V f (t) is the current following speed, a f (t) is the current following vehicle acceleration, D f (t) is the current following distance, V r (t) is the relative speed to the preceding vehicle, T w (t) is the average waiting time of vehicles on the road section, and Delay(t) is the average delay time of vehicles on the road section;
[0099] S22, construct the action space a(t)∈[-3,3](m / s 2 ),
[0100] The following safety distance constraint is
[0101] S23. Design the reward function. The specific steps are as follows:
[0102] The total reward value is
[0103] R cf =w safety ×R safety +weffi ×R effi +w comf ×R comf +w fuel ×R fuel
[0104] Where R safety is the security bonus, R effi is the efficiency reward, R comf For comfort bonus, R fuel For fuel consumption reward, w safety ,w effi ,w comf w fuel They are the weights of the corresponding rewards, specifically:
[0105] (1) Security Rewards
[0106] R safety =D f -D s
[0107] D f The intelligent agent calculates the safe distance that will not cause a collision within the driver's reaction time based on its own following speed and the speed of the preceding vehicle. s The following distance maintained by the current agent;
[0108] (2) Efficiency Rewards
[0109] R effi =A×R speed +B×R totalwait +C×R totaldelay
[0110] Where A, B, and C are the weights of each reward, which can be further divided into three factors: (a) the deviation of the following vehicle speed before and after the action; (b) the numerical difference of the total waiting time of the road section before and after the action; (c) the numerical difference of the total vehicle delay before and after the action.
[0111] a) Deviation of the following vehicle speed before and after the action
[0112] R speed =V f (t+1)-V f (t)
[0113] V f (t) is the following speed of the agent before taking action, V f (t+1) is the following speed after adopting the following action;
[0114] b) Section waiting time
[0115] R totalwait=TW total (t)-TW total (t+1)
[0116] TW total (t) is the total waiting time before the agent selects an action, TW total (t+1) is the total waiting time of the road section after the agent selects an action;
[0117] c) The difference in the total delay of vehicles on the road section before and after the action
[0118] R totaldelay =Delay(t)-Delay(t+1)
[0119] Delay(t) is the total delay of the road section before the agent adopts the action, and Delay(t+1) is the total delay of the road section after adopting the following action;
[0120] (3) Comfort Reward
[0121]
[0122] jerk and jerk′ are the comfort parameters before and after the action execution, respectively;
[0123] (4) Fuel consumption reward
[0124] R fuel =P′ fuel -P fuel
[0125] P fuel ,P′ fuel They are the fuel consumption before and after the action is executed;
[0126] S24. Build a DDPG-based network framework to train the agent's car-following decision:
[0127] (1) Initialize the Critic evaluation network structure Q(s,a|θ) Q ) and Actor decision network structure μ(s|θ μ ) and its weight θ Q and θ μ , initialize the Critic target - evaluate the network structure Q′(s′,a′|θ Q′ ) and Actor goal-decision network structure μ′(s′|θ μ′ ) and its weight θ Q′ and θ μ′ , and iterate step (2) in each subsequent step;
[0128] (2) Initialize a random process K for action exploration; initialize state s1 as the first state of the current state sequence, obtain its feature network, and iterate steps (3) to (10) in each subsequent step;
[0129] (3) According to the current state s t Strategy and exploration noise K t Select the following acceleration action a t =μ(s t |θ μ )+K t ;
[0130] (4) Vehicle performs action a t And observe the reward r t And observe the new state s t+1 ;
[0131] (5) The interaction data of the intelligent autonomous driving vehicle (s t ,a t ,r t ,s t+1 ) is stored in the experience pool;
[0132] (6) Sample random mini-batches of N transformations in the replay experience pool;
[0133] (7) The Critic evaluation network calculates the current Q value y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ );
[0134] (8) Update the critic evaluation network parameters by minimizing the loss. The loss value is expressed as
[0135]
[0136] (9) Use the sampled policy gradient to update the gradient of the Actor policy network;
[0137] (10) Update the parameters θ of the Critic target-evaluation network and the Actor target-decision network Q′ and θ μ′ .
[0138] refer to Figure 3 , step S3 comprises the following steps:
[0139] Construct a Markov reward process for a specific autonomous driving method:
[0140] S31. Constructing the state space Where a is the vehicle acceleration, v is the vehicle speed, d is the relative distance between the agent and the surrounding vehicles, (l,f,lf,ll,rf,rl) are the front vehicle, the rear vehicle, the left front vehicle, the left rear vehicle, the right front vehicle, and the right rear vehicle, respectively. lon ,x lat is the position of the agent, Lane_index is used to indicate the lane number, and Occ is the occupancy rate of the target road section;
[0141] S32, constructing an action space a(t) = [-1, 0, 1], where -1 indicates changing lanes to the right non-dedicated lane, 0 indicates keeping the current lane, and 1 indicates changing lanes to the left dedicated lane;
[0142] S33. Design a reward function. The reward function and the car-following model have some repeated calculation methods. The specific steps are as follows:
[0143] The total reward value is
[0144] R lc =w safety ×R safety +w effi ×R effi +w comf ×R comf +w fuel ×R fuel +w adv ×R adv
[0145] w safety ,w effi ,w comf ,w fuel ,w adv They are the weights of the corresponding rewards, specifically:
[0146] (1) Security Rewards
[0147]
[0148] Specify penalties and rewards for collisions and non-collisions respectively;
[0149] (2) Efficiency Rewards
[0150] R effi =-|v agent -v desired |
[0151] Among them, v agent is the current speed of the agent, v desired is the expected speed, set to the road speed limit;
[0152] (3) Comfort Reward
[0153]
[0154] The setting method is the same as the corresponding reward setting method of the car-following model;
[0155] (4) Fuel consumption reward
[0156] R fuel =P′ fuel -P fuel
[0157] The setting method is the same as the corresponding reward setting method of the car-following model;
[0158] (5) Lane Change Advantage Reward
[0159] R adv =A×R Lane_index +B×R Occ
[0160] Where A and B are the weights of each reward, which can be further divided into two factors: (a) the reward or punishment for whether the driverless vehicle chooses the dedicated lane; (b) the change in lane occupancy before and after the action;
[0161] a) Rewards and penalties for autonomous vehicles choosing dedicated lanes
[0162]
[0163] Reward or punish driverless vehicles based on whether they choose dedicated lanes;
[0164] b) Lane occupancy changes before and after the action
[0165] R Occ =logOcc(t)-logOcc(t+1)
[0166] Occ(t) and Occ(t+1) are the occupancy rates of the road section before and after the agent takes the action;
[0167] S34. Build a DQN-based network framework to train the lane-changing decision of the intelligent agent:
[0168] DQN selects the best action based on the Q table, where a dynamic ε-greedy exploration strategy is selected. The learning probability ε value decreases with the increase in the number of iterations. A random number is selected during iteration. If it is greater than ε, the action with the largest Q value is selected. If it is less than ε, random exploration is performed. This makes the strategy selection biased towards the exploration mode in the early stage of learning and biased towards the utilization mode in the later stage of learning. Among them: ε′ is the subsequent learning probability, ε initial ,ε finalare the initial and final ε respectively; considering the situation where the target vehicle cannot change lanes, that is, the target vehicle changes lanes from the left lane to the left or from the right lane to the right, the Q value of the unexecutable strategy in the current state is eliminated to avoid invalid exploration when the maximum Q value is selected.
[0169] Furthermore, the specific implementation steps of the aforementioned step S4 are:
[0170] In the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road. When making a lane changing decision, if the lane changing strategy is executed, the status of the leading vehicle is updated. If the lane changing strategy is not executed, a following decision is made to control the following vehicle acceleration to achieve vehicle trajectory optimization control.
[0171] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A vehicle trajectory optimization control method for an autonomous driving lane, characterized in that: The following steps are involved: S1. Design application scenarios of dedicated lanes under mixed traffic flow conditions; S2. Design the Markov reward process of the car-following model, determine the state space, action space and reward value, build a network framework based on the deep deterministic policy gradient algorithm (DDPG), and train the car-following decision of the intelligent agent; S3. Design the Markov reward process of the lane-changing model, determine the state space, action space and reward value, build a network framework based on the deep reinforcement learning algorithm (DQN), and train the lane-changing decision of the intelligent agent; S4. Under the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road to achieve vehicle trajectory optimization control.
2. The vehicle trajectory optimization control method for an autonomous driving dedicated lane according to claim 1, characterized in that: The application scenarios of dedicated lanes under mixed traffic flow conditions include: multi-lane traffic environment, which includes autonomous driving dedicated lanes and non-autonomous driving dedicated lanes.
3. The vehicle trajectory optimization control method for an autonomous driving dedicated lane according to claim 1, characterized in that: In step S2, a Markov reward process is constructed for a specific autonomous driving method, including the following steps: S21, construct state space s(t) = {V f (t),a f (t),D f (t),V r (t),T w (t), Delay(t)}, where V f (t) is the current following vehicle speed, a f (t) is the current following vehicle acceleration, D f (t) is the current following distance, V r (t) is the relative speed to the preceding vehicle, T w (t) is the average waiting time of vehicles on the road section, and Delay(t) is the average delay time of vehicles on the road section; S22, construct the action space a(t)∈[-3,3](m / s 2 ), The following safety distance constraint is S23. Design the reward function. The specific steps are as follows: The total reward value is R cf =w safety ×R safety +w effi ×R effi +w comf ×R comf +w fuel ×R fuel Where R safety is the security bonus, R effi is the efficiency reward, R comf For comfort bonus, R fuel For fuel consumption reward, w safety ,w effi ,w comf ,w fuel They are the weights of the corresponding rewards, specifically: (1) Security Rewards R safety =D f -D s D f The intelligent agent calculates the safe distance that will not cause a collision within the driver's reaction time based on its own following speed and the speed of the preceding vehicle. s The following distance maintained by the current agent; (2) Efficiency Rewards R effi =A×R speed +B×R totalwait +C×R totaldelay Where A, B, and C are the weights of each reward, which can be further divided into three factors: (a) the deviation of the following vehicle speed before and after the action; (b) the numerical difference of the total waiting time of the road section before and after the action; (c) the numerical difference of the total vehicle delay before and after the action; a) Deviation of the following vehicle speed before and after the action R speed =V f (t+1)-V f (t) V f (t) is the following speed of the agent before taking action, V f (t+1) is the following speed after adopting the following action; b) Section waiting time R totalwait =TW total (t)-TW total (t+1) TW total (t) is the total waiting time before the agent selects an action, TW total (t+1) is the total waiting time of the road section after the agent selects an action; c) The difference in the total delay of vehicles on the road section before and after the action R totaldelay =Delay(t)-Delay(t+1) Delay(t) is the total delay of the road section before the agent adopts the action, and Delay(t+1) is the total delay of the road section after adopting the following action; (3) Comfort Reward jerk and jerk′ are the comfort parameters before and after the action execution, respectively; (4) Fuel consumption reward R fuel =P′ fuel -P fuel P fuel ,P′ fuel They are the fuel consumption before and after the action is executed; S24. Build a DDPG-based network framework to train the agent's car-following decision: (1) Initialize the Critic evaluation network structure Q(s,a|θ) Q ) and Actor decision network structure μ(s|θ μ ) and its weight θ Q and θ μ , initialize the Critic target - evaluate the network structure Q′(s′,a′|θ Q′ ) and Actor goal-decision network structure μ′(s′|θ μ′ ) and its weight θ Q′ and θ μ′ , and iterate step (2) in each subsequent step; (2) Initialize a random process K for action exploration; initialize state s1 as the first state of the current state sequence, and obtain its feature network. And iterate steps (3) to (10) in each subsequent step; (3) According to the current state s t Strategy and exploration noise K t Select the following acceleration action a t =μ(s t |θ μ )+K t ; (4) Vehicle performs action a t And observe the reward r t And observe the new state s t+1 ; (5) The interaction data of the intelligent autonomous driving vehicle (s t ,a t ,r t ,s t+1 ) is stored in the experience pool; (6) Sample random mini-batches of N transformations in the replay experience pool; (7) The Critic evaluation network calculates the current Q value y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ); (8) Update the critic evaluation network parameters by minimizing the loss. The loss value is expressed as (9) Use the sampled policy gradient to update the gradient of the Actor policy network; (10) Update the parameters θ of the Critic target-evaluation network and the Actor target-decision network Q′ and θ μ′ .
4. The vehicle trajectory optimization control method for an autonomous driving dedicated lane according to claim 1, characterized in that: In step S3, a Markov reward process is constructed for a specific autonomous driving method, including the following steps: S31. Constructing the state space Where a is the vehicle acceleration, v is the vehicle speed, d is the relative distance between the agent and the surrounding vehicles, (l,f,lf,ll,rf,rl) are the front vehicle, the rear vehicle, the left front vehicle, the left rear vehicle, the right front vehicle, and the right rear vehicle, respectively. lon ,x lat is the position of the agent, Lane_index is used to indicate the lane number, and Occ is the occupancy rate of the target road section; S32, constructing an action space a(t) = [-1, 0, 1], where -1 indicates changing lanes to the right non-dedicated lane, 0 indicates keeping the current lane, and 1 indicates changing lanes to the left dedicated lane; S33. Design a reward function. The reward function and the car-following model have some repeated calculation methods. The specific steps are as follows: The total reward value is R lc =w safety ×R safety +w effi ×R effi +w comf ×R comf +w fuel ×R fuel +w adv ×R adv w safety ,w effi ,w comf ,w fuel ,w adv They are the weights of the corresponding rewards, specifically: (1) Security Rewards Specify penalties and rewards for collisions and non-collisions respectively; (2) Efficiency Rewards R effi =-|v agent -v desired | Among them, v agent is the current speed of the agent, v desired is the expected speed, set to the road speed limit; (3) Comfort Reward The setting method is the same as the corresponding reward setting method of the car-following model; (4) Fuel consumption reward R fuel =P′ fuel -P fuel The setting method is the same as the corresponding reward setting method of the car-following model; (5) Lane Change Advantage Reward R adv =A×R Lane_index +B×R Occ Where A and B are the weights of each reward, which can be further divided into two factors: (a) the reward or punishment for whether the driverless vehicle chooses the dedicated lane; (b) the change in lane occupancy before and after the action; a) Rewards and penalties for autonomous vehicles choosing dedicated lanes Reward or punish driverless vehicles based on whether they choose dedicated lanes; b) Lane occupancy changes before and after the action R Occ =logOcc(t)-logOcc(t+1) Occ(t) and Occ(t+1) are the occupancy rates of the road section before and after the agent takes the action; S34. Build a DQN-based network framework to train the lane-changing decision of the agent, and select the quintic polynomial for the lane-changing trajectory: DQN selects the best action based on the Q table, where a dynamic ε-greedy exploration strategy is selected. The learning probability ε value decreases with the increase in the number of iterations. A random number is selected during iteration. If it is greater than ε, the action with the largest Q value is selected. If it is less than ε, random exploration is performed. This makes the strategy selection biased towards the exploration mode in the early stage of learning and biased towards the utilization mode in the later stage of learning. Among them: ε′ is the subsequent learning probability, ε initial ,ε final are the initial and final ε respectively; considering the situation where the target vehicle cannot change lanes, that is, the target vehicle changes lanes from the left lane to the left or from the right lane to the right, the Q value of the unexecutable strategy in the current state is eliminated to avoid invalid exploration when the maximum Q value is selected.
5. The vehicle trajectory optimization control method for an autonomous driving dedicated lane according to claim 1, characterized in that: The specific implementation steps of step S4 are: In the dedicated lane setting scenario designed in step S1, the following model trained in step S2 and the lane changing model trained in step S3 are applied to the target road. When making a lane changing decision, if the lane changing strategy is executed, the status of the leading vehicle is updated. If the lane changing strategy is not executed, a following decision is made to control the following vehicle acceleration to achieve vehicle trajectory optimization control.
Citation Information
Cited By
Interactive test method and device for automatic driving model, medium and product
CN119337738A
Intelligent cooperative control method, system and equipment for highway ramps and medium
CN121171044A
