A deep reinforcement learning vehicle platoon control method based on supervised learning

Abstract

The present invention discloses a deep reinforcement learning vehicle platoon control method based on guided learning. First, an adaptive cruise control model is introduced as a guide in the initial training of a reinforcement learning agent, and an action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. In the later training process, a reasonable reward mechanism is designed to incentivize the agent to gradually reduce or even ignore the output of the guide, so that the agent can learn a policy function that is better than the initial guidance through action exploration. The designed reinforcement learning algorithm does not require complex vehicle dynamics modeling, but only relies on the designed observation space to find the optimal policy function under the incentive of the reward function, highlighting the advantages of the model-free algorithm. Finally, under the premise of ensuring safe decision-making, the designed deep reinforcement learning algorithm based on guided learning can train the agent to make a control strategy that is more reasonable and more human-like than traditional control strategies.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a deep reinforcement learning vehicle queue control method. Background Art

[0002] Intelligent connected vehicles (ICVs) are equipped with advanced onboard sensors, controllers, actuators, and other devices, integrating modern communication and networking technologies to enable intelligent information exchange and sharing between vehicles and the X (people, vehicles, roads, and the cloud). They possess capabilities such as complex environmental perception, intelligent decision-making, and coordinated control. With continuous breakthroughs in perception, decision-making, and communication technologies, ICVs are gradually moving toward practical application. Their widespread adoption will have a disruptive impact on road safety, traffic management, and travel efficiency. In a connected environment, autonomous vehicles adjust their longitudinal motion to form a platoon, achieving consistent speeds and desired spacing, thus forming a vehicle platoon. Platooning offers numerous advantages: Platooning can reduce the spacing between vehicles in a platoon, improve road capacity, and alleviate traffic congestion to a certain extent. Real-time sharing of status information among platoon vehicles facilitates rapid and accurate decision-making, improving road safety. Since energy consumption during vehicle travel is proportional to air resistance, platooning can reduce air resistance, lower vehicle emissions and fuel consumption, and improve the urban environment.

[0003] Methods for implementing vehicle platooning primarily include cooperative adaptive cruise control, model predictive control, and optimal control. However, these methods suffer from slow response times, are unable to rapidly handle sudden road conditions, and are overly dependent on the communication topology and system model between vehicles in the platoon. Reinforcement learning, an experience-driven approach for sequential decision-making, can be applied to control decisions in vehicle platooning systems. However, training traditional reinforcement learning agents to achieve human-like control requires considerable time and computational effort. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention provides a deep reinforcement learning vehicle platoon control method based on guided learning. First, an adaptive cruise control model is introduced as a guide in the initial training of the reinforcement learning agent, and an action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. In the later training process, a reasonable reward mechanism is designed to incentivize the agent to gradually reduce or even ignore the output of the guide, so that the agent can learn a policy function that is better than the initial guidance through action exploration. The designed reinforcement learning algorithm does not require complex vehicle dynamics modeling and only relies on the designed observation space to find the optimal policy function under the incentive of the reward function, highlighting the advantages of the model-free algorithm. Finally, under the premise of ensuring safe decision-making, the designed deep reinforcement learning algorithm based on guided learning can train the agent to make control strategies that are more reasonable and more human-like than traditional control strategies.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: N intelligent connected vehicles form a vehicle platoon. The dynamic model of the vehicles in the platoon is described as follows:

[0007]

[0008] Where i = 0, 1, ..., N represents the i-th vehicle; x0(t) and v0(t) represent the position and velocity of the pilot vehicle, respectively. des_0 (t) represents the acceleration of the pilot vehicle, and its change is given by humans; x i (t), v i (t) and a des_i (t), i = 1, ..., N are the position, velocity, and expected acceleration of the i-th following vehicle in the vehicle queue, respectively, where the expected acceleration of the following vehicle is the action output of the reinforcement learning agent in the decision-making process; a actual_i (t) represents the actual acceleration of the vehicle, τ is the time constant of the vehicle engine;

[0009] Step 2: Build the vehicle platoon system environment;

[0010] Relative distance between vehicles d real_i The actual position of the preceding vehicle minus the actual position of the vehicle itself is obtained as follows:

[0011] d real_i =x i-1 (t)-x i (t) (2)

[0012] Define the expected distance between vehicles as d safe_i :

[0013] dsafe_i =D default +v ego_i *t gap (3)

[0014] Among them D default is the distance between vehicles when they are stationary, v ego_i is the vehicle speed, t gap is the headway;

[0015] Define the spacing error d err_i for:

[0016] d err_i =d safe_i -d real_i (4)

[0017] The desired speed v of the following vehicle des_i Defined as follows: If d err_i Greater than zero, that is, d real_i Less than d safe_i , the expected speed v of the following vehicle des_i Given is the speed v of the vehicle ahead ego_i-1 and set speed v set The minimum value of the two, the rear vehicle and the front vehicle maintain a certain safe distance; if d real_i Greater than d safe_i , the expected speed v of the following vehicle des_i Then the given speed v set ,Right now:

[0018]

[0019] The velocity error is defined as follows:

[0020] v err_i =v des_i -v ego_i (6)

[0021] Step 3: Design a guidance-based deep reinforcement learning algorithm;

[0022] Step 3-1: Design of the instructor;

[0023] Introducing an adaptive cruise control model as a guide for training reinforcement learning agents, including speed control mode and distance control mode;

[0024] For the i-th following vehicle, in speed control mode, the expected acceleration of the intelligent connected vehicle is as follows:

[0025] a vc_i =K v *(v tar_i -v ego_i ) (7)

[0026] Among them, K v is the speed control gain, v tar_i is the expected speed of the i-th following vehicle;

[0027] In the spacing control mode, the expected acceleration of the i-th following vehicle is given by the following formula:

[0028] a sc_i =K v *v pf_i -K d *d err_i (8)

[0029] Among them, K d is the control gain of the spacing; v pf_i =v ego_i-1 -v ego_i is the speed difference between the vehicle in front and the vehicle in front;

[0030] Taking into account the speed control and distance control of the adaptive cruise control model, the expected acceleration of the i-th following vehicle is given as follows:

[0031]

[0032] Step 3-2: Deep Deterministic Policy Gradient (DDPG) algorithm;

[0033] The DDPG algorithm uses four artificial neural networks: Actor network, Critic network, Actor target network, and Critic target network to approximate the behavior policy function and the target policy function;

[0034] The Actor network is used to approximate the behavior policy μ by making the decision network parameters θ μ The output of the parameterized Actor network is:

[0035] a t =μ(s t |θ μ ) (10)

[0036] Among them, s t represents the current state, θ μ represents the Actor network parameters under strategy μ, μ(s t |θ μ ) represents the network parameter θ μ Under the condition, the agent is in the current state s t Decision output;

[0037] Based on the OU process, the DDPG algorithm adds Gaussian noise to the action output:

[0038]

[0039] Where N t noise representing motion exploration;

[0040] The critic network is parameterized as θ Q , used to approximate the value function:

[0041]

[0042] The data format stored in the experience pool is: (s t ,a t ,r t ,s t+1 ), r t represents the immediate reward after the agent makes a decision, s t+1 Represents the state that the agent transfers to after making a decision; extract small batches of data from the experience pool to train and update the parameters of the above four networks; the critic network updates its parameters according to the following loss function:

[0043] y i =r i +γQ'(s i+1 ,μ'(s i+1 |θ μ ')|θ Q ') (13)

[0044]

[0045] Among them, y i represents the target return value, r i represents the immediate reward in the current state, γ represents the discount factor, s i+1 Indicates the next state of the current state, θ μ′ are the target policy network parameters, θ Q′ is the target value network parameter, μ'(s i+1 |θ μ ') represents the target output action, Q'(s i+1 ,μ'(s i+1 |θ μ ')|θ Q ') represents the target action value;

[0046] According to the gradient of the expected target J, the Actor network is updated as follows:

[0047]

[0048] in, represents the gradient of the action-value function with respect to the action, Represents the policy μ to policy network parameter θ μ gradient;

[0049] After the parameters of the Actor network and the Critic network are updated, the parameters of the corresponding two target networks are soft-updated according to formula (16):

[0050]

[0051] Among them, τ is the soft update parameter;

[0052] Step 3-3: Design the action synthesizer:

[0053] The adaptive cruise control model is introduced as a guide for the training of reinforcement learning agents, and an action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. The fusion mechanism is shown in the following formula:

[0054] u S (t)=λ*u C (t)+(1-λ)*u A (t) (17)

[0055] Among them, u S (t)=(a s_1 ,a s_2 ,...) T is the action value vector after fusion, u C (t)=(a c_1 ,a c_2 ,...) T is the output vector of the director, u A (t)=(a A_1 ,a A_2 ,...) T is the decision action vector of the reinforcement learning agent, and λ is the weight value;

[0056] Step 4: Design the action space, state space, and reward function of the guidance-based deep reinforcement learning algorithm;

[0057] The action space column vector is defined as A = (a A_1 ,a A_2 ,...,a A_i ,...,λ) T , where a A_i are the output accelerations of the N-1 following vehicles;

[0058] For each of the N-1 following vehicles in the system, three state observation inputs are selected; for the i-th following vehicle, the vehicle speed v is selected. ego_i , speed error v with the preceding vehicle err_i and the integral of the velocity error ∫verr_i As the state observation of the i-th following car; add the output value a of the director to the system state observation; c_i and the agent's decision action value a A_i The difference a E_i , that is, a E_i =a c_i -a A_i ; The entire system state observation vector is s:

[0059]

[0060] The vehicle speed error v is selected from three aspects: stability, comfort, and following efficiency. err_i , spacing error d err_i , vehicle acceleration a A_i The reward function of the indicator design system;

[0061] Step 5: Adjust the algorithm training hyperparameters in Table 1 and train the DDPG agent based on supervised learning to output the optimal control strategy.

[0062] Table 1: DDPG algorithm training hyperparameters

[0063]

[0064] Preferably, τ=0.001.

[0065] The beneficial effects of the present invention are as follows:

[0066] In the decision-making and control method of the present invention, first, an adaptive cruise control model is introduced as a guide in the early stages of reinforcement learning agent training, and an action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. The purpose is to improve the efficiency of the reinforcement learning agent in learning the optimal strategy, overcome the problem of slow algorithm convergence, and reduce the catastrophic consequences that may occur in the early stages of intelligent connected vehicles training. In the later training process, a reasonable reward mechanism is designed to incentivize the agent to gradually reduce or even ignore the output of the guide, so that the agent can learn a policy function that is better than the initial guidance through action exploration, thereby optimizing the algorithm efficiency and saving a large amount of training time and computing power costs. Secondly, the designed reinforcement learning algorithm does not require complex vehicle dynamics modeling, but only relies on the designed observation space to find the optimal policy function under the incentive of the reward function, highlighting the advantages of the model-free algorithm. Finally, under the premise of ensuring safe decision-making, the designed deep reinforcement learning algorithm based on guidance can train the agent to make a control strategy that is more reasonable and more human-like than traditional control strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1This is a deep reinforcement learning vehicle platoon control method framework based on guided learning in the present invention.

[0068] Figure 2 This is a comparison chart of the first 100 training steps of the unsupervised and guided agent training in the present invention: (a) the reward change trend of decision training using the unsupervised DDPG algorithm, (b) the reward change trend of decision training using the guided DDPG algorithm.

[0069] Figure 3 In this paper, in a scenario where the acceleration of the pilot vehicle changes sinusoidally, the agent uses a guidance-based deep reinforcement learning algorithm. After learning a better strategy in the early stage of training: Figure 3 (a) Follow the speed change trend of the vehicle, Figure 3 (b) Following the trend of the relative distance between vehicles; after the agent finally learns the optimal strategy under the incentive of the reward function: Figure 3 (c) Follow the speed change trend of the vehicle, Figure 3 (d) Follow the trend of the relative distance between vehicles. DETAILED DESCRIPTION

[0070] The present invention will be further described below with reference to the accompanying drawings and examples.

[0071] like Figure 1 As shown in FIG, a deep reinforcement learning vehicle platoon control method based on guided learning includes the following steps:

[0072] Step 1: Design a platoon of N intelligent connected vehicles on a straight highway. The dynamic model of the vehicles in the platoon is described as follows:

[0073]

[0074] Where i = 0, 1, ..., N represents the i-th vehicle; x0(t) and v0(t) represent the position and velocity of the pilot vehicle, respectively. des_0 (t) represents the acceleration of the pilot vehicle, and its change is given by humans; x i (t), v i (t) and a des_i (t), i = 1, ..., N are the position, speed, and expected acceleration of the i-th following vehicle in the vehicle queue; the expected acceleration of the following vehicle is the action output in the decision-making process of the intelligent agent.

[0075] Step 2: Build the vehicle platoon system environment.

[0076] Relative distance between vehicles d real_i The actual position of the preceding vehicle minus the actual position of the vehicle itself is obtained as follows:

[0077] d real_i =x i-1 (t)-x i (t) (2)

[0078] Define the expected distance between vehicles as d safe_i :

[0079] d safe_i =D default +v ego_i *t gap (3)

[0080] where v ego_i is the vehicle speed, t gap is the headway, D default is the distance between vehicles when they are stationary;

[0081] Define the spacing error d err_i for:

[0082] d err_i =d safe_i -d real_i (4)

[0083] The desired speed v of the following vehicle des_i Defined as follows: If d err_i Greater than zero, that is, d real_i Less than d safe_i , the expected speed v of the following vehicle des_i Given is the speed v of the vehicle ahead ego_i-1 and the controller sets the speed v set The minimum of the two; in this way, the rear car can maintain a certain safe distance from the front car; if d real_i Greater than d safe_i , the expected speed v of the following vehicle des_i The given speed v is set by the controller set ,Right now:

[0084]

[0085] The velocity error is defined as follows:

[0086] v err_i =v des_i -v ego_i (6)

[0087] Speed ​​error v err_i It will be used as a type of state observation input to the system during agent training.

[0088] Step 3: Design an instruction-based deep reinforcement learning algorithm:

[0089] Step 3.1 Design of the instructor:

[0090] An adaptive cruise control model is introduced as a guide for the initial training of the agent, which includes speed control mode and distance control mode. For the i-th following vehicle, in speed control mode, the expected acceleration of the intelligent connected vehicle is as follows:

[0091] a vc_i =K v *(v tar_i -v ego_i ) (7)

[0092] Among them, K v is the speed control gain, v tar_i is the expected speed of the i-th following vehicle;

[0093] In the spacing control mode, the expected acceleration of the i-th following vehicle is given by the following formula:

[0094] a sc_i =K v *v pf_i -K d *d err_i (8)

[0095] Among them, v pf_i =v ego_i-1 -v ego_i is the speed difference between the vehicle and the preceding vehicle; d err_i is the spacing error between the desired spacing and the actual spacing of the vehicle. Its calculation method is shown in formulas (3) and (4).

[0096] Taking into account the speed control and distance control of the adaptive cruise control model, the expected acceleration of the i-th following vehicle is given as follows:

[0097]

[0098] Step 3.2 Deep Deterministic Policy Gradient (DDPG) algorithm:

[0099] The control goal for vehicle platooning is to enable vehicles to form a platoon through mutual communication and collaboration, maintaining a consistent speed and desired inter-vehicle spacing. Furthermore, performance indicators such as following efficiency and energy efficiency should be considered. From a reinforcement learning perspective, vehicle platoon control is a multi-objective decision-making problem with a high-dimensional continuous state space and action space. The DDPG algorithm is suitable for making decisions in this high-dimensional continuous action space and state space, and can perform multi-objective decision planning based on a reward function, meeting the control requirements of vehicle platooning.

[0100] The DDPG algorithm uses four artificial neural networks (Actor network, Critic network, Actor target network, Critic target network) to approximate the behavior policy function and the target policy function. The Actor network of the DDPG algorithm is used to approximate the behavior policy μ, and the decision network parameter θ is used to determine the behavior policy μ. μ The output of the parameterized Actor network is:

[0101] a t =μ(s t |θ μ ) (10)

[0102] Based on the Ohrnstein-Uhlenbeck Process (OU process), the DDPG algorithm adds Gaussian noise to the action output to enable the agent to better explore the unknown environment:

[0103]

[0104] Where N t Gaussian noise representing action exploration.

[0105] The critic network is parameterized as θ Q , used to approximate the value function:

[0106]

[0107] The data format stored in the experience playback buffer is: (s t ,a t ,r t ,s t+1 ), extract small batches of data from the experience pool to train and update the parameters of the above four networks; the critic network updates its parameters according to the following loss function:

[0108] y i =r i +γQ'(s i+1 ,μ'(s i+1 |θ μ ')|θ Q ') (13)

[0109]

[0110] According to the gradient of the expected target J, the Actor network is updated as follows:

[0111]

[0112] In order to improve the stability of network training, after the parameters of the Actor network and the Critic network are updated, the parameters of the corresponding two target networks are soft-updated according to formula (16):

[0113]

[0114] Among them, τ is the soft update parameter, generally τ=0.001, θ μ′ are the target policy network parameters, θ Q′ is the target value network parameter. The main mechanism of soft update is: instead of directly replacing the parameter, it slowly updates it with the parameter τ. This ensures that the target network parameter is less affected by the update of the evaluation network parameter in each iteration, and does not produce excessive changes. This, to a certain extent, reduces the fluctuation of the value function and policy gradient iterative calculation.

[0115] Step 3.3 Design the action synthesizer:

[0116] The traditional DDPG algorithm requires a lot of ineffective exploration during the training process before it can slowly learn the appropriate control strategy. The algorithm is inefficient and the model converges slowly. Therefore, the adaptive cruise control model is introduced as a guide in the early stage of reinforcement learning agent training. An action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. The fusion mechanism is shown in the following formula:

[0117] u S (t)=λ*u C (t)+(1-λ)*u A (t) (17)

[0118] Among them, u S (t)=(a s_1 ,a s_2 ,...) T is the action value vector after fusion, u C (t)=(a c_1 ,a c_2 ,...) T is the output vector of the director, u A (t)=(a A_1 ,a A_2 ,...) T is the decision action vector of the reinforcement learning agent, and λ is the weight value. The weight value is one of the decision actions of the reinforcement learning agent and is obtained by the agent through self-learning based on appropriate system state observations.

[0119] Step 4: Design the action space, state space, and reward function for the guidance-based deep reinforcement learning algorithm:

[0120] In the vehicle platoon network, we use the case of N = 4 to describe the action space of the system. The column vector of the action space is defined as A = (a A_1 ,a A_2 ,a A_3 ,λ) T , where a A_i (i=1, 2, 3) are the output accelerations of the three following vehicles respectively, and λ is the weight value of the action synthesizer, which also serves as a decision output of the intelligent agent.

[0121] For each of the three following vehicles in the system, three state observation inputs are selected. The following example is given for the i-th following vehicle: ego_i , speed error v with the preceding vehicle err_i and the integral of the velocity error ∫v err_i As the state observation of the i-th following car. Because the adaptive cruise control model is introduced as the instructor for the initial training of the agent, and in order to make the agent better learn the weight value of the action synthesizer, the output value of the instructor a is added to the system state observation C_i and the agent's decision action value a A_i The difference a E_i , that is, a E_i =a C_i -a A_i , i=1,2,3. The entire system state observation vector is s:

[0122]

[0123] The reward function is a very important part of the reinforcement learning algorithm design. The ultimate goal of reinforcement learning is to find an optimal strategy that enables the agent to obtain the most rewards in the interaction with the environment. There is no fixed paradigm for the design of the reward function. Any explicit logical representation that can optimize the agent's decision-making can be used as part of the reward function. Its design method depends largely on the initial training results of the agent and the experience of the researcher. This paper mainly selects the vehicle speed error v from three aspects: stability, comfort, and vehicle following efficiency. err_i , spacing error d err_i , vehicle acceleration a A_i Design the reward function of the system based on indicators such as

[0124] The primary goal of vehicle platooning is to ensure that the following vehicle can maintain the same speed as the leading vehicle at the desired distance. Therefore, taking the first following vehicle as an example, the first part of the reward function is designed as follows:

[0125]

[0126] Among them, v err_1is the speed error of the first following car. The smaller the speed error, the greater the reward value. A_1 is the acceleration of the first following vehicle. During the platooning process, if the acceleration of the following vehicle does not fluctuate and is as close to zero as possible, the system will receive a larger reward value. This can also reflect the stability and comfort of the platooning process to a certain extent. err_1 is the relative distance error, i.e., the difference between the actual relative distance between the first following vehicle and the preceding vehicle and the expected distance; M1 and K1 are the additional rewards when the speed error and relative distance error reach a certain range, respectively. The specific settings are as follows:

[0127]

[0128]

[0129] The reward function design for the remaining following vehicles is similar to that for the first following vehicle:

[0130]

[0131]

[0132] During the platooning process, no vehicle is allowed to collide or reverse, so the reward function in the second part is designed as follows:

[0133]

[0134] Among them, when the relative distance d between the following vehicles appears rel_i <0 or the actual speed v of the following vehicle ego_i When <0, is_done is set to true, indicating that a collision or reversing occurs during the vehicle platoon. The current training step needs to be terminated immediately and a very large negative reward is obtained from the system environment. This is to warn the agent that such actions are very dangerous and should be avoided as much as possible during training.

[0135] Because the adaptive cruise control model is introduced to guide the agent's early training, it is hoped that after the agent reaches a relatively good decision-making level through certain guidance, it will gradually reduce or even ignore the guidance of the instructor in the middle and late stages of training. Through the agent's own exploration and the incentive of the reward function, it will be able to learn a policy function that is superior to the early guidance. Therefore, the reward function in the third part is designed based on the weight value λ in the action synthesizer:

[0136]

[0137] Among them, the larger the λ is, the stronger the guidance effect of the director, and the smaller the reward the agent obtains from the system environment at this time; conversely, the smaller the λ is, the weaker the guidance effect of the director, and the agent is more inclined to its own exploration actions, while also being able to obtain greater rewards from the system environment; ε is a bias term that prevents the extreme case of infinite reward function when λ = 0.

[0138] Combining the reward functions of the above three parts, the final reward function design for agent learning is given as follows:

[0139] R=k*(r1+r2+r3+r4+r5) (26)

[0140] Among them, k is a global system parameter that can adjust the range of variation of the global reward obtained by the agent from the environment during simulation training, which helps to analyze the decision-making level of the agent as a whole.

[0141] Step 5: According to the hyperparameters given in Table 1, train the DDPG reinforcement learning agent based on supervised learning to output the optimal control policy.

[0142] Table 1: DDPG algorithm training hyperparameters

[0143]

[0144] In the numerical simulation of vehicle platooning, the simulation initialization conditions are set as follows: the initial speed of the pilot vehicle is 25m / s, the initial position is 100m, and the acceleration of the pilot vehicle shows a sinusoidal variation trend: a(t) = Asin(0.2*t), A = 0.5m / s 2 The initial speeds of the following vehicles are 27m / s, 25m / s, and 26m / s, respectively; and their initial positions are 70m, 35m, and 0m, respectively.

[0145] In the same simulation environment, the unsupervised algorithm and the guided algorithm are used to train the agent's decision-making. The hyperparameter settings of the agent training are shown in Table 1. The training results of the two algorithms in the first 100 training steps are shown in Table 1. Figure 2 As shown in the figure, the circled line represents the cumulative reward value of the current training step, reflecting the quality of the agent's single-step decision; the asterisk line represents the average reward value calculated for a certain number of the latest training steps, reflecting the changing trend of the agent's overall decision-making level.

[0146] from Figure 2 (a) It can be seen that in the first 100 training steps, the agents without early guidance are all in invalid exploration, and the decision-making actions of the agents all end the current training step early by crashing or reversing; Figure 2In (b), the agent makes decisions with the help of the director and begins to converge to a better strategy around the 30th training step. At this time, the actions output by the action synthesizer can basically complete the entire training step and obtain a good reward return. After that, under the motivation of the reward function, the agent explores the action space with higher rewards through the action exploration process.

[0147] It can be seen that the proposed guidance-based deep reinforcement learning algorithm can use the instructor to make the agent's decision-making actions at a better level in the early stage of the agent's training, and explore and learn the optimal strategy based on the better strategy, which greatly shortens the agent's ineffective exploration process, greatly improves the algorithm efficiency, and saves computing and time costs.

[0148] Figure 3 Shows the changing trends in the speed and relative spacing of following vehicles when training an agent to make decisions using a guidance-based algorithm. Figure 3 (a) In the early stages of training, after the agent learns the optimal strategy based on guidance, there is a certain lag between the speed changes of the following vehicles, but the overall decision-making effect is good, and the entire training step can be completed without crashing or reversing. Figure 3 (c) shows that in the later stage of training, under the motivation of the reward function, the agent explores the action space with higher returns and learns the optimal strategy. The speed of the following car can well follow the speed changes of the leading car, with almost no lag, showing good real-time response performance. Figure 3 (b) and 3(d), the optimal decision finally learned by the agent is also better than the previous better strategy in maintaining the relative distance between vehicles. This shows that the agent finally learned a policy function that is better than the previous guidance under the motivation of the reward function.

Claims

1. A deep reinforcement learning vehicle platoon control method based on supervised learning, characterized in that: The following steps are involved: Step 1: N intelligent connected vehicles form a vehicle platoon. The dynamic model of the vehicles in the platoon is described as follows: Where i = 0, 1, ..., N represents the i-th vehicle; x0(t) and v0(t) represent the position and velocity of the pilot vehicle, respectively. des_0 (t) represents the acceleration of the pilot vehicle, and its change is given by humans; x i (t), v i (t) and a des_i (t), i = 1, ..., N are the position, velocity, and expected acceleration of the i-th following vehicle in the vehicle queue, respectively, where the expected acceleration of the following vehicle is the action output of the reinforcement learning agent in the decision-making process; a actual_i (t) represents the actual acceleration of the vehicle, τ is the time constant of the vehicle engine; Step 2: Build the vehicle platoon system environment; Relative distance between vehicles d real_i The actual position of the preceding vehicle minus the actual position of the vehicle itself is obtained as follows: d real_i =x i-1 (t)-x i (t) (2) Define the expected distance between vehicles as d safe_i : d safe_i =D default +v ego_i *t gap (3) Among them D default is the distance between vehicles when they are stationary, v ego_i is the vehicle speed, t gap is the headway; Define the spacing error d err_i for: d err_i =d safe_i -d real_i (4) The desired speed v of the following vehicle des_i Defined as follows: If d err_i Greater than zero, that is, d real_i Less than d safe_i , the expected speed v of the following vehicle des_i Given is the speed v of the vehicle ahead ego_i-1 and set speed v set The minimum value of the two, the rear vehicle and the front vehicle maintain a certain safe distance; if d real_i Greater than d safe_i , the expected speed v of the following vehicle des_i Then the given speed v set ,Right now: The velocity error is defined as follows: v err_i =v des_i -v ego_i (6) Step 3: Design a guidance-based deep reinforcement learning algorithm; Step 3-1: Design of the instructor; Introducing an adaptive cruise control model as a guide for training reinforcement learning agents, including speed control mode and distance control mode; For the i-th following vehicle, in speed control mode, the expected acceleration of the intelligent connected vehicle is as follows: a vc_i =K v *(v tar_i -v ego_i ) (7) Among them, K v is the speed control gain, v tar_i is the expected speed of the i-th following vehicle; In the spacing control mode, the expected acceleration of the i-th following vehicle is given by the following formula: a sc_i =K v *v pf_i -K d *d err_i (8) Among them, K d is the control gain of the spacing; v pf_i =v ego_i-1 -v ego_i is the speed difference between the vehicle and the preceding vehicle; Taking into account the speed control and distance control of the adaptive cruise control model, the expected acceleration of the i-th following vehicle is given as follows: Step 3-2: Deep Deterministic Policy Gradient (DDPG) algorithm; The DDPG algorithm uses four artificial neural networks: Actor network, Critic network, Actor target network, and Critic target network to approximate the behavior policy function and the target policy function; The Actor network is used to approximate the behavior policy μ by making the decision network parameters θ μ The output of the parameterized Actor network is: a t =μ(s t |θ μ ) (10) Among them, s t represents the current state, θ μ represents the Actor network parameters under strategy μ, μ(s t |θ μ ) represents the network parameter θ μ Under the condition, the agent is in the current state s t Decision output; Based on the OU process, the DDPG algorithm adds Gaussian noise to the action output: Where N t noise representing motion exploration; The critic network is parameterized as θ Q , used to approximate the value function: The data format stored in the experience pool is: (s t ,a t ,r t ,s t+1 ), r t represents the immediate reward after the agent makes a decision, s t+1 Represents the state that the agent transfers to after making a decision; extract small batches of data from the experience pool to train and update the parameters of the above four networks; the critic network updates its parameters according to the following loss function: Among them, y i represents the target return value, r i represents the immediate reward in the current state, γ represents the discount factor, s i+1 Indicates the next state of the current state, θ μ′ are the target policy network parameters, θ Q′ is the target value network parameter, μ'(s i+1 |θ μ' ) represents the target output action, Q'(s i+1 ,μ'(s i+1 |θ μ' )|θ Q' ) represents the target action value; According to the gradient of the expected target J, the Actor network is updated as follows: in, represents the gradient of the action-value function with respect to the action, Represents the policy μ to policy network parameter θ μ gradient; After the parameters of the Actor network and the Critic network are updated, the parameters of the corresponding two target networks are soft-updated according to formula (16): Among them, τ is the soft update parameter; Step 3-3: Design the action synthesizer: The adaptive cruise control model is introduced as a guide for the training of reinforcement learning agents, and an action synthesizer is designed to fuse the action output of the reinforcement learning agent with the output of the guide. The fusion mechanism is shown in the following formula: u S (t)=λ*u C (t)+(1-λ)*u A (t) (17) Among them, u S (t)=(a s_1 ,a s_2 ,...) T is the action value vector after fusion, u C (t)=(a c_1 ,a c_2 ,...) T is the output vector of the director, u A (t)=(a A_1 ,a A_2 ,...) T is the decision action vector of the reinforcement learning agent, and λ is the weight value; Step 4: Design the action space, state space, and reward function of the guidance-based deep reinforcement learning algorithm; The action space column vector is defined as A = (a A_1 ,a A_2 ,...,a A_i ,...,λ) T , where a A_i are the output accelerations of the N-1 following vehicles; For each of the N-1 following vehicles in the system, three state observation inputs are selected; for the i-th following vehicle, the vehicle speed v is selected. ego_i , speed error v with the preceding vehicle err_i and the integral of the velocity error ∫v err_i As the state observation of the i-th following car; add the output value a of the director to the system state observation; c_i and the agent's decision action value a A_i The difference a E_i , that is, a E_i =a c_i -a A_i ; The entire system state observation vector is s: The vehicle speed error v is selected from three aspects: stability, comfort, and following efficiency. err_i , spacing error d err_i , vehicle acceleration a A_i The reward function of the indicator design system; Step 5: Tune the algorithm training hyperparameters and train the supervised learning DDPG agent to output the optimal control strategy. Hyperparameter η A Represents the learning rate of the Actor network, the hyperparameter η C represents the learning rate of the Critic network, the hyperparameter M represents the experience pool size, the hyperparameter α represents the batch size, the hyperparameter γ represents the discount factor, and the hyperparameter T e Indicates the maximum training step, hyperparameter represents the target smoothing factor, hyperparameter T f Indicates the maximum training step for a single scene.

2. The deep reinforcement learning vehicle platoon control method based on supervised learning according to claim 1 is characterized in that: The τ=0.001.

Images