Deep reinforcement learning-based vehicle platoon control method and system

Abstract

A deep reinforcement learning-based vehicle platoon control method and system. The method comprises: on the basis of a vehicle dynamics model, constructing a motion simulation model for vehicle platoon operation; on the basis of the motion simulation model for vehicle platoon operation and vehicle operating states, by using a constant time headway strategy, obtaining a state space and an action space for vehicle platoon control; and establishing a DDPG network on the basis of the state space and the action space for vehicle platoon control, using a reward function mechanism to train the DDPG network, stopping training when the total reward obtained per episode during the training process no longer increases, so as to obtain a trained control strategy network, and by using the trained control strategy network, performing platoon control.

Description

A vehicle platoon control method and system based on deep reinforcement learning

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202311084199.6, filed with the China Patent Office on August 25, 2023, entitled “A Vehicle Queue Control Method and System Based on Deep Reinforcement Learning,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention belongs to the field of vehicle intelligent decision-making control, and specifically relates to a vehicle queue control method and system based on deep reinforcement learning. Background Art

[0004] In recent years, with the continuous development of communications and computer technologies, intelligent transportation systems have gradually become one of the key solutions to traffic problems. Intelligent connected vehicles, in particular, have become a global automotive development trend, offering advantages such as reducing traffic congestion, improving road utilization, and reducing energy consumption. In an intelligent connected environment, autonomous vehicles can form a platoon by adjusting their longitudinal motion, achieving consistent speeds and desired spacing between vehicles, thus forming a vehicle platoon. Vehicle platooning offers numerous advantages: it can improve road capacity, enhance road safety, reduce fuel consumption, and mitigate environmental pollution.

[0005] Platoon control based on the PID control method does not require the construction of a platoon model and can adapt to different scenarios. However, it requires manual setting of control parameters, and a single set of control parameters often cannot meet the needs of all traffic scenarios. Platoon control based on the MPC model predictive control method requires the construction of an accurate vehicle platoon model and requires continuous optimization calculations to obtain the control variable. This computationally intensive and complex method makes it difficult to meet the requirements of vehicle control with high real-time requirements.

[0006] Summary of the Invention

[0007] The purpose of the present invention is to provide a vehicle platoon control method and system based on deep reinforcement learning to overcome the problems of low vehicle platoon control accuracy and complex calculation process in the existing technology.

[0008] A vehicle platoon control method based on deep reinforcement learning includes the following steps:

[0009] S1, constructing a motion simulation model of vehicle platooning based on the vehicle dynamics model;

[0010] S2, based on the motion simulation model of the vehicle platoon and the vehicle operation status, uses the constant time interval strategy to obtain the state space and action space of the vehicle platoon control;

[0011] In step S3, a DDPG network is established based on the state space and action space of the vehicle platoon control. The DDPG network is trained using a reward function mechanism until the total reward obtained in each round of training stops increasing. The trained control policy network is obtained and used to perform platoon control.

[0012] Preferably, the vehicle adopts the following dynamic model, which is specifically expressed as follows:

[0013] The continuous dynamic model is discretized by Euler formula, and the discretization results are as follows: i (t+T)=p i (t)+T*v i (t) v i (t+T)=v i (t)+T*a i (t)

[0014] where p i (t), v i (t), a i (t) respectively represent the position, velocity and acceleration of vehicle i at time t, u i (t) represents the control input of vehicle i, τ represents the dynamic constant of the vehicle transmission system, and T is the sampling time of the control system.

[0015] Preferably, the vehicle platoon control of the motion simulation model satisfies the constraints: min ≤v i (t)≤v max u min ≤u i (t)≤u max

[0016] where v max =25m / s,v min =0m / s, is the vehicle speed constraint, u max =3m / s 2 ,u min =-3m / s 2 , which are the constraints for vehicle control input.

[0017] Preferably, obtaining the state space and action space of the vehicle platoon control specifically includes: designing the state space S according to the distance between vehicles, the speed difference, the speed of vehicle i and the acceleration of vehicle i, as shown below: S = {Δp i,i-1 ,Δv i,i-1 ,v i ,a i}

[0018] The action space is A=u i (t), represents the control input of vehicle i.

[0019] Preferably, the reward function mechanism adopts the form of reward function R as follows: R = r1 + k2r2 + k3r3 + k4r4

[0020] where r1 = k 11 |Δd i,t |+k 12 |Δv i,t |, r1 represents the control efficiency, which is the weighted sum of the absolute value of the distance error and the absolute value of the speed difference. i,t →0 and Δv i,t →0, r1→0; k 11 ,k 12 Represents the weighting coefficient.

[0021] Preferably, r2 represents driving comfort. When the acceleration is greater, the driving comfort is lower.

[0022] r3 represents the collision penalty, Δp i,t When it is a negative value, the smaller it is, the more serious the collision is, and the greater the penalty is. In other cases, it is 0.

[0023] r4 represents speed penalty, which is imposed when the speed exceeds the speed constraint; v min and v max Indicates the minimum and maximum values ​​of the speed constraint, which are 0m / s and 25m / s respectively;

[0024] k2, k3, k4 are reward weight coefficients.

[0025] Preferably, a pilot car and a follower car are constructed in the motion simulation model, and the follower car is the control car; the trajectory of the pilot car is set by itself, and the movement of the pilot car is determined according to the training round. When the training starts, the pilot car is in a stationary state. When the control car moves normally in a stationary state, the pilot car changes to a uniform speed, uniformly variable speed or variable speed movement; the initial state of the control car is randomly generated when the constraints are met; when the distance error between the two cars is greater than the set threshold, a training round ends.

[0026] Preferably, the updating process of the control strategy network is as follows: V(s) is the value of evaluating the current state, which is the mean of the state-action values ​​of all actions in the current state, and can be approximated as the state-action value of the current state through the Monk Carlo approximation;

[0027] The control strategy network updates the network parameters by maximizing L(θ) as follows:

[0028] The Target network uses a soft update method, and the update method is as follows: ω′=ωτ+ω′(1-τ),θ′=θτ+θ′(1-τ)

[0029] Where τ∈[0,1], here we take τ=0.01;

[0030] All network parameters are continuously updated during the training process until the total reward of the round no longer increases, and the training is completed.

[0031] A vehicle platoon control system based on deep reinforcement learning, including a simulation module, a state acquisition module and a control module:

[0032] The simulation module builds a motion simulation model of vehicle platooning based on the vehicle dynamics model;

[0033] The state acquisition module uses a constant time spacing strategy to obtain the state space and action space of vehicle platoon control based on the motion simulation model of the vehicle platoon and the vehicle operation status;

[0034] The control module establishes a DDPG network based on the state space and action space of vehicle platoon control and trains the DDPG network using a reward function mechanism until the total reward obtained in each round of training stops increasing. The trained control policy network is obtained and used to perform platoon control.

[0035] Preferably, the vehicle adopts the following dynamic model, which is specifically expressed as follows:

[0036] The continuous dynamic model is discretized by Euler formula, and the discretization results are as follows: i (t+T)=p i (t)+T*v i (t) v i (t+T)=v i (t)+T*a i (t)

[0037] where p i (t), v i (t), a i (t) respectively represent the position, velocity and acceleration of vehicle i at time t, u i (t) represents the control input of vehicle i, τ represents the dynamic constant of the vehicle transmission system, and T is the sampling time of the control system.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects:

[0039] The present invention discloses a vehicle queue control method based on deep reinforcement learning. The method constructs a motion simulation model of the vehicle queue operation based on the vehicle dynamics model. Based on the motion simulation model of the vehicle queue operation and the vehicle operation status, a constant time interval strategy is used to obtain the state space and action space of the vehicle queue control. A DDPG network is established based on the state space and action space of the vehicle queue control, and the DDPG network is trained using a reward function mechanism. Training is stopped until the total reward obtained in each round during training no longer increases, thereby obtaining a trained control strategy network, which is used to control the queue. The present invention takes multiple factors into consideration to achieve queue control. After the control strategy of the queue control method is trained, complex calculations are not required during execution, resulting in very high control efficiency. Moreover, the control effect can be maintained under different conditions of the current scene without the need for parameter changes.

[0040] The reward function mechanism is used to comprehensively consider multiple factors including driving efficiency, comfort, collision penalty and speed penalty, and the constant distance safety distance strategy is adopted to ensure the safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a flow chart of a vehicle queue control method based on a deep neural network in an embodiment of the present invention.

[0042] FIG2 is a schematic diagram of a vehicle platoon communication topology structure according to an embodiment of the present invention.

[0043] FIG3 is a diagram showing vehicle state changes of a vehicle queue in an initial traffic scenario according to an embodiment of the present invention.

[0044] FIG4 is a diagram showing vehicle state changes in a vehicle queue in a trapezoidal disturbance traffic scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] As shown in FIG1 , the present invention provides a vehicle queue control method based on deep reinforcement learning, which specifically includes the following steps:

[0048] S1, constructing a motion simulation model of vehicle platooning based on the vehicle dynamics model;

[0049] The vehicle adopts the following dynamic model, which is specifically expressed as follows:

[0050] The continuous dynamic model is discretized by Euler formula, and the discretization results are as follows: i (t+T)=p i (t)+T*v i (t) v i (t+T)=v i (t)+T*a i (t)

[0051] where p i (t), v i (t), a i (t) respectively represent the position, velocity and acceleration of vehicle i at time t, u i(t) represents the control input of vehicle i, τ represents the dynamic constant of the vehicle transmission system, and T is the sampling time of the control system.

[0052] In order to ensure the safety of vehicle driving, the vehicle platoon control complies with the following constraints: min ≤v i (t)≤v max u min ≤u i (t)≤u max (3)

[0053] where v max =25m / s,v min =0m / s, is the vehicle speed constraint, u max =3m / s 2 ,u min =-3m / s 2 , which are the constraints for vehicle control input.

[0054] S2, based on the motion simulation model of vehicle platoon operation, uses a constant time spacing strategy according to the vehicle operation status to obtain the state space and action space of vehicle platoon control;

[0055] The spacing strategy of the vehicle queue is determined using a constant time spacing strategy. The expected spacing between vehicle i and the preceding vehicle i-1 is:

[0056] where h i ,l i is a given parameter, where h i represents the expected time interval between vehicle i and the preceding vehicle, l i Indicates the safe distance between vehicle i and the vehicle ahead when it is stationary.

[0057] The distance and speed difference between vehicle i and the preceding vehicle i-1 are defined as follows: Δp i,i-1 (t) = p i-1 (t)-p i (t)-L Δv i,i-1 (t) = v i-1 (t)-v i (t) (5)

[0058] where p i-1 (t), p i (t) represents the position of vehicle i-1 and vehicle i, respectively, v i-1 (t), v i (t) represents the speed of the preceding vehicle i-1 and vehicle i, respectively, and L represents the length of the vehicle.

[0059] The distance error between vehicle i and the preceding vehicle i-1 is the vehicle spacing minus the expected spacing, which can be expressed as:

[0060] The communication topology of Figure 2 is used, and the state space S is designed according to the distance between vehicles, the speed difference, the speed of vehicle i, and the acceleration of vehicle i, as shown below: S = {Δp i,i-1 ,Δv i,i-1 ,v i ,a i} (7)

[0061] The action space is A=u i (t), represents the control input of vehicle i.

[0062] S3, design a reward function mechanism for platoon control. The reward function in the reward function mechanism includes multiple factors such as control efficiency, driving comfort, collision penalty, and speed penalty.

[0063] The form of the reward function R in the reward function mechanism is as follows: R=r1+k2r2+k3r3+k4r4 (8)

[0064] where r1 = k 11 |Δd i,t |+k 12 |Δv i,t |, r1 represents the control efficiency, which is the weighted sum of the absolute value of the distance error and the absolute value of the speed difference. i,t →0 and Δv i,t →0, r1→0; k 11 ,k 12 represents the weighting coefficient;

[0065] r2 represents driving comfort. When the acceleration is greater, the driving comfort is lower.

[0066] r3 represents the collision penalty, Δp i,t When it is a negative value, the smaller it is, the more serious the collision is, and the greater the penalty is. In other cases, it is 0.

[0067] r4 represents speed penalty, which is imposed when the speed exceeds the speed constraint; v min and v max Indicates the minimum and maximum values ​​of the speed constraint, which are 0m / s and 25m / s respectively;

[0068] k2, k3, k4 are reward weight coefficients.

[0069] S4: Establish a DDPG network based on the state space and action space of the vehicle platoon control, and train the established DDPG network until the total reward obtained in each round during training no longer increases, thus obtaining a trained control strategy network.

[0070] DDPG is a deterministic policy gradient algorithm, a deterministic deep reinforcement learning algorithm based on the deterministic policy method of the Actor-Critic structure. The deterministic policy is the output of the control policy network, which is the vehicle's action. The Actor-Critic is the executor-judge. The Actor executes the action, and the Critic judges the current state and action.

[0071] The DDPG network consists of a value network, Critic, and a policy network, Actor. The value network is used to evaluate the value of the current state and action, and the policy network is used to decide the execution action based on the current state.

[0072] 4.1: Constructing the DDPG network, which is based on the Actor-Critic structure. The Actor network is used to perform actions and is the policy network π θ (s), Critic is used to evaluate the current state and action, which is the value network Q ω (s,u).

[0073] Then construct Target-Actorπ with the same structure as the value network and strategy network respectively θ′ (s) and Target-CriticQ ω′ (s,u) network. It is used to reduce the correlation between data when training the network and speed up the training process.

[0074] Where s represents the current state, u represents the action u=π θ (s), θ, ω are the parameters of the Actor and Critic networks, and θ', ω' are the parameters of the Target-Actor and Target-Critic networks.

[0075] 4.2: Train the DDPG network. Train the DDPG network according to the reward function until the total reward of the round no longer increases. Then end the training process and finally obtain the trained policy network.

[0076] In the motion simulation model, a pilot car and a follower car are constructed. The follower car is the control car. The pilot car's trajectory is set by the pilot car, and its motion is determined based on the training rounds. Initially, the pilot car is stationary. Once the control car can move normally from a stationary state, the pilot car switches to a constant speed, a constant variable speed, or a variable speed. This progressively more challenging approach expedites the training process.

[0077] The initial state of the control car is randomly generated while satisfying the constraints. When the distance error between the two cars is greater than the set threshold (30m), the training round ends and the next training round begins again.

[0078] According to the previously designed state space, the vehicle parameters in Table 1 and the real-time state of the vehicle obtained by the motion simulation model, the state s at each time step can be obtained. i (t)={Δp i,i-1 (t),Δv i,i-1 (t),v i (t),a i (t)}.

[0079] According to the previously designed reward function, the vehicle parameters in Table 1 and the real-time state of the vehicle obtained by the motion simulation model, the state R of each time step can be obtained i (t).

[0080] The policy network Actor is based on the current state s i (t) and the exploration mechanism can obtain the action u that should be performed in the current state i (t).

[0081] Table 1

[0082] Exploration mechanism, which requires continuous exploration during the training process, uses a normal distribution selection method for actions, as shown below: t =N(u t =π θ (s t ),δ 2 ) (9)

[0083] The mean of the normal distribution is u t =π θ (s t )According to the action obtained by the policy network, the variance is δ 2 , and the variance decreases as the training progresses, δ 2 The update method of δ is as follows: 2 =δ 2 *α (10)

[0084] Among them, α is less than 1, and the variance becomes smaller and smaller as the training progresses, that is, as the training progresses, the exploration intensity becomes smaller and smaller.

[0085] Current status i (t) and action u i (t) After execution in the motion simulation model, the next state s of the current state can be obtained i(t+1). And the state s obtained in each round i (t), action u i (t) and the next state s i (t+1) and the reward function R i (t) is stored in the buffer as training data. A total of D pieces of data can be stored, and m records are selected for training each time.

[0086] The updating process of the value network. Q(s t ,u t ;ω) is used to evaluate the current state s t and action u t , through the Critic Network Q ω Got it.

[0087] Q(s t+1 ,u t+1 ;ω′) is used to evaluate the next state s t+1 and the next action u t+1 , through the Target-Critic network Q ω′ The next action is based on the next state s t+1 Through Target-Actor Network π θ′ Got it.

[0088] In this way, the evaluation network of the current state action and action is different from the evaluation network of the next state and next action, which can reduce the correlation and improve stability.

[0089] The value network continuously updates the network parameters by minimizing J(ω). The update method is as follows:

[0090] Among them, γ∈[0,1] is the attenuation factor.

[0091] Update process of the control strategy network: V(s) is the value of evaluating the current state, which is the mean value of the state action of all actions in the current state. It can be approximated to the value of the state action of the current state through Monk Carlo approximation.

[0092] The control strategy network updates the network parameters by maximizing L(θ) as follows:

[0093] The Target network adopts a soft update method, and the update method is as follows: ω′=ωτ+ω′(1-τ),θ′=θτ+θ′(1-τ) (13)

[0094] Where τ∈[0,1], here we take τ=0.01.

[0095] All network parameters are continuously updated during the training process until the total reward of the round no longer increases, and the training is completed.

[0096] S5, use the trained control strategy network to perform queue control;

[0097] The trained dynamic strategy network can be used to calculate the current vehicle state s i (t), we can get the optimal action that should be performed in the current state, that is, u i * (t)=π θ (s i (t)). And no longer requires an exploration mechanism as in the training process, and according to the control input constraints, u i * (t)=clip(u i * (t),u min ,u max ), that is, the control input must be within the control constraint range. When the control input exceeds the range, the control input is the nearest constraint amount.

[0098] Example:

[0099] A vehicle platoon consisting of a leader vehicle and four follower vehicles was constructed. Different scenarios were designed to use the policy network to control the vehicle platoon, including initialization simulation scenarios and vehicle platoon perturbation simulation scenarios, to observe the control performance of the policy network.

[0100] In a simulation scenario for an initial platoon, assume that the lead vehicle travels at 25 m / s and the following vehicles travel at random initial positions and velocities. As shown in Figure 3, after a period of adjustment, the position and velocity errors between the vehicles gradually decrease to zero, and the distance between the vehicles becomes consistent and their speeds become the same. This verifies that the control strategy enables optimal coordinated driving in the platoon under the initial simulation scenario.

[0101] Set up a disturbance simulation scenario for a vehicle platoon. Assume that the vehicle platoon moves in coordination at a uniform speed of 5 m / s at the initial speed, and the pilot vehicle moves at a speed of 1.5 m / s after 30 seconds. 2 Start to accelerate, start to run at a constant speed at 40s, run for a while, and then at 60s, start to run at a constant speed of -1.5m / s 2 The vehicle decelerates, returning to a speed of 5 m / s after 70 seconds before continuing at a constant speed. As shown in Figure 4, when the lead vehicle begins to accelerate or decelerate, the following vehicles are able to promptly adapt to the changes in the leading vehicle's state, adjusting their own speed and acceleration. The spacing between the vehicles approaches the desired spacing, and the speeds of the vehicles converge. This demonstrates that the control strategy ensures that the following vehicles consistently follow the state changes of the leading vehicle and maintains the stability of the vehicle platoon.

Claims

1. A vehicle platoon control method based on deep reinforcement learning, characterized in that: The following steps are involved: Construct a motion simulation model of vehicle platoon operation based on the vehicle dynamics model; Based on the motion simulation model of vehicle platoon operation and the vehicle operation status, the constant time spacing strategy is used to obtain the state space and action space of vehicle platoon control; A DDPG network is established based on the state space and action space of vehicle queue control, and the reward function mechanism is used to train the DDPG network until the total reward obtained in each round during training no longer increases. The training is stopped to obtain a trained control strategy network, which is used to perform queue control.

2. The vehicle platoon control method based on deep reinforcement learning according to claim 1, characterized in that: The vehicle adopts the following dynamic model, which is specifically expressed as follows: The continuous dynamic model is discretized by Euler formula, and the discretization results are as follows: p i (t+T)=p i (t)+T*v i (t) v i (t+T)=v i (t)+T*a i (t) where p i (t), v i (t), a i (t) represent the position, velocity and acceleration of vehicle i at time t, u i (t) represents the control input of vehicle i, τ represents the dynamic constant of the vehicle transmission system, and T is the sampling time of the control system.

3. The vehicle queue control method based on deep reinforcement learning according to claim 2, characterized in that: The vehicle platoon control of the motion simulation model satisfies the constraints: v min ≤v i (t)≤v max u min ≤u i (t)≤u max where v max =25m / s,v min =0m / s, is the vehicle speed constraint, u max =3m / s 2 ,u min =-3m / s 2 , which are the constraints for vehicle control input.

4. The vehicle queue control method based on deep reinforcement learning according to claim 1, characterized in that: The state space and action space of the vehicle platoon control are obtained by designing the state space S according to the distance between vehicles, the speed difference, the speed of vehicle i and the acceleration of vehicle i, as shown below: S={Δp i,i-1 ,Δv i,i-1 ,v i ,a i } The action space is A = u i (t), represents the control input of vehicle i.

5. The vehicle queue control method based on deep reinforcement learning according to claim 1, characterized in that: The reward function mechanism adopts the form of reward function R as follows: R = r1 + k2r2 + k3r3 + k4r4 Where r1 = k 11 |Δd i,t |+k 12 |Δv i,t |, r1 represents the control efficiency, which is the weighted sum of the absolute value of the distance error and the absolute value of the speed difference. i,t →0 and Δv i,t →0, r1→0; k 11 ,k 12 Represents the weighting coefficient.

6. The vehicle queue control method based on deep reinforcement learning according to claim 5, characterized in that: r2 represents driving comfort. When the acceleration is greater, the driving comfort is lower. r3 represents the collision penalty, Δp i,t When it is a negative value, the smaller it is, the more serious the collision is, and the greater the penalty is. In other cases, it is 0; r4 represents speed penalty, which is imposed when the speed exceeds the speed constraint; v min and v max Indicates the minimum and maximum values ​​of the speed constraint, which are 0m / s and 25m / s respectively; k2, k3, k4 are reward weight coefficients.

7. The vehicle queue control method based on deep reinforcement learning according to claim 1, characterized in that: In the motion simulation model, a pilot car and a follower car are constructed, and the follower car is the control car. The trajectory of the pilot car is set by itself, and the movement of the pilot car is determined according to the training rounds. At the beginning of training, the pilot car is stationary. When the control car moves normally in a stationary state, the pilot car changes to uniform speed, uniformly variable speed or variable speed movement. The initial state of the control car is randomly generated when the constraints are met. When the distance error between the two cars is greater than the set threshold, a training round ends.

8. The vehicle queue control method based on deep reinforcement learning according to claim 1, characterized in that: Update process of control strategy network: V(s) is the value of evaluating the current state, which is the mean value of the state action of all actions in the current state. It can be approximated as the value of the state action of the current state through Monk Carlo approximation; The control strategy network updates the network parameters by maximizing L(θ) as follows: The Target network uses a soft update method. The update method is as follows: ω′=ωτ+ω′(1-τ), θ′=θτ+θ′(1-τ) Where τ∈[0,1], here we take τ=0.01; All network parameters are continuously updated during the training process until the total reward of each round no longer increases. When , the training is completed.

9. A vehicle platoon control system based on deep reinforcement learning, characterized in that: Including simulation module, status acquisition module and control module: The simulation module builds a motion simulation model of vehicle platoon operation based on the vehicle dynamics model; The state acquisition module uses a constant time spacing strategy to acquire the state space and action space of the vehicle platoon control based on the motion simulation model of the vehicle platoon operation and the vehicle operation status; The control module establishes a DDPG network based on the state space and action space of vehicle queue control, and trains the DDPG network using the reward function mechanism until the total reward obtained in each round during training no longer increases, and then stops training to obtain a trained control strategy network, which is used to perform queue control.

10. The vehicle platoon control system based on deep reinforcement learning according to claim 9, characterized in that: The vehicle adopts the following dynamic model, which is specifically expressed as follows: The continuous dynamic model is discretized by Euler formula, and the discretization results are as follows: p i (t+T)=p i (t)+T*v i (t) v i (t+T)=v i (t)+T*a i (t) where p i (t), v i (t), a i (t) represent the position, velocity and acceleration of vehicle i at time t, u i (t) represents the control input of vehicle i, τ represents the dynamic constant of the vehicle transmission system, and T is the sampling time of the control system.

Images