A deep reinforcement learning platoon control method considering vehicle heterogeneity

Abstract

The application discloses a kind of deep reinforcement learning platoon control method considering vehicle heterogeneity.The steps of the present application are:1, a three-order dynamic model that can be used to identify the dynamic model parameters of heterogeneous vehicles is established;2, when the vehicle platoon is moving, the vehicle collects its own motion state data through sensors, and uses the least square method to identify the transmission system time parameter τ in real time online according to the collected motion state;3, a following controller for the following vehicle is designed using the PPO algorithm;4, the following vehicle obtains the motion state information of the adjacent preceding vehicle through V2V communication;5, using the motion state information of the adjacent preceding vehicle, the following controller obtains the expected acceleration, cooperates with the string stability and a first-order filter to suppress acceleration shock, obtains the real acceleration, and further controls the vehicle to follow the vehicle and realizes the goal of stable driving of the vehicle platoon.The present application is suitable for various motion conditions and has better optimization effect.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and provides a deep reinforcement learning-based formation control method that takes into account the heterogeneity of vehicles in heterogeneous fleets with different V2V communication. Specifically, it involves a heterogeneous fleet formation control optimization technology based on the deep reinforcement learning PPO algorithm. Background Technology

[0002] As a crucial component of intelligent transportation systems, intelligent connected vehicles can acquire real-time information on surrounding vehicles and nearby roads through advanced onboard sensors and V2X (Vehicle-to-Everything) communication technology, enabling information exchange between vehicles and between vehicles and infrastructure. Cooperative Adaptive Cruise Control (CACC), based on V2V (Vehicle-to-Vehicle) communication, achieves rapid and precise intelligent control, reducing traffic accidents caused by human error, enhancing vehicle active safety, simplifying traffic control and management, and effectively alleviating road congestion.

[0003] Building upon CACC, combining it with deep reinforcement learning-based platooning control can improve the stability and safety of heterogeneous platoons. Despite extensive research on CACC, a problem remains when applying this technology to a networked platoon of autonomous vehicles. Most strategies do not consider platoon dynamics parameters, making them difficult to apply to diverse heterogeneous platoons, and their scalability is also limited. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a deep reinforcement learning-based formation control method that considers vehicle heterogeneity. This method mainly comprises three parts: fleet dynamics modeling, online model parameter identification, and deep reinforcement learning-based formation control model design, achieving stability and safety for heterogeneous fleets. Furthermore, following vehicles exhibit good following stability and safety under varying speeds and accelerations of the lead vehicle.

[0005] The specific design of this invention is carried out according to the following steps:

[0006] Step 1: Establish a third-order dynamic model that can be used to identify the dynamic model parameters of heterogeneous vehicles:

[0007]

[0008] Since the actual dynamic model is unknown, and considering the heterogeneity of the fleet, the vehicle dynamic parameters need to be obtained, we built a third-order dynamic model to identify this important vehicle dynamic parameter.

[0009] Where, p n，t v n，t a n，t , These represent the position, velocity, acceleration, and control input of vehicle n at time t, respectively, where Δt is the sampling interval, and τ is the input. n，t and φ n，t Let be the transmission system time parameters and input delay of vehicle n at time t. This indicates the control input after eliminating input delay.

[0010] Furthermore, using the CTH spacing strategy, the following error dynamic characteristics of vehicle n can be modeled as follows:

[0011]

[0012] Among them, e n，p，t and e n，v，t e n，a，t Let D be the distance error, speed error, and acceleration error of vehicle n relative to vehicle n-1 at time t. n，t Let be the total unknown disturbance in the following error dynamics model of vehicle n at time t, and its expression is as follows:

[0013]

[0014] After obtaining the following error dynamic model for a single vehicle, the platoon dynamic model becomes the following error dynamic model for all following vehicles in the platoon. Furthermore, the transmission system time constant... Input delay φ n，t and external disturbances D n，t These are the parameters of the vehicle dynamics model, D n，t Unable to obtain, this can be overcome by leveraging the generalization ability of reinforcement learning models.

[0015] Step 2: When the convoy is moving, the vehicles collect their own motion state data through sensors. Based on the collected motion state, the least squares method is used to identify the transmission system time parameter τ in real time online. The specific operation is as follows:

[0016] To make the reinforcement learning formation control model applicable to different heterogeneous vehicle fleets, it is proposed to use a cache memory approach to uniformly set the input delay of all vehicles to the maximum input delay φ of all vehicles in the fleet. m .

[0017] Since the time parameters of the transmission system are time-varying, we propose to use the least squares method with an adaptive forgetting factor to identify the transmission system time parameter τ in the dynamic model of step 1 in real time online. n，tHere, we first present the Laplace transform transfer function G(s) from the control input to the acceleration in the dynamic model as follows:

[0018]

[0019] Where s is the complex variable after the Laplace transform.

[0020] Considering that the least squares method with adaptive forgetting factor is not suitable for parameter identification of time-delay systems, we propose to use the t-φ-th method. m The control input at time t and the acceleration at time t are taken as an input-output data pair and subjected to a new Laplace transform. Thus, the transfer function to be identified is transformed into:

[0021]

[0022] The parametric equations are to be obtained using the first-order backward difference method, as follows:

[0023] y n，k =φ n，k θ n，k

[0024] Among them, y n，k =a n，k , Then, the identification parameters are to be obtained using the following recursive formula.

[0025]

[0026] Among them, K k P represents the intermediate parameter at time t. k-1 λ represents the intermediate parameter at time t-1. k-1 Let t represent the forgetting factor at time t-1, and k represent the time k.

[0027] To further improve the accuracy of parameter identification, the forgetting factor λ k Adaptive adjustment is achieved through the following formula:

[0028]

[0029] In the above formula, e base , round(), and δ∈[0,1] represent the baseline error, rounding function, and sensitivity coefficient, respectively. λ min λ k The lower limit of change, ∈ k Intermediate parameters

[0030] Step 3: Design a following controller for the following vehicle using the PPO algorithm. Details are as follows:

[0031] The state s of car n at time t n，t The proposed definition is as follows:

[0032]

[0033] in, o_delay represents an intermediate variable. n，t Indicates communication delay. Indicates the identification parameters of the adjacent preceding vehicle. τ represents the control input data within the delay time window, indicating the input delay. n，t Represents its own identification parameters

[0034] The reward function for the follower controller is:

[0035]

[0036] Where, α 1，t α 2,t α 3，t These are the time-varying weights to be designed, used to dynamically determine the relative importance of minimizing the following speed error, control input, and rate of change of acceleration. n，p，max e n，v，max , and jerk max These represent the maximum allowable values ​​for following distance error, following speed error, control input, and rate of change of acceleration, respectively. (Rate of change of acceleration jerk) n，t The proposed definition is as follows:

[0037]

[0038] in, This represents the control input of the nth vehicle at time t. This represents the parameter tau obtained through identification.

[0039] Step 4: The following vehicle obtains the motion status information of the adjacent preceding vehicle through V2V communication. The motion status information includes the position p of the preceding vehicle n-1. n-1，t Speed ​​v n-1，t acceleration a n-1，t Its own transmission system time parameters Considering communication latency, the following vehicle obtains its own transmission system time parameters for identification of the preceding vehicle as follows:

[0040] Step 5: Using the information from the vehicle in front in Step 4, the following controller in Step 3 calculates the desired acceleration, thereby controlling the vehicle to follow the vehicle stably and achieve steady state of the convoy.

[0041] Step 6: Repeat steps 4 to 5 continuously during the driving process to ensure that each vehicle in the convoy can maintain stable driving and achieve adaptive control optimization for heterogeneous convoys.

[0042] Compared with the prior art, the advantages of the present invention are as follows:

[0043] This invention employs a third-order dynamics model, adding a vehicle inertial element, which is closer to reality than the traditional second-order vehicle model. A first-order acceleration filter is used in the follower controller, mitigating acceleration oscillations to some extent. This invention can be applied to platoons of heterogeneous vehicles, greatly expanding its applicability. The control method proposed in this invention does not rely on the object model or environmental information, thus compensating for the shortcomings of current model-based methods to some extent. This invention is applicable to various motion conditions, and can safely and comfortably achieve platoon stability under various rapid acceleration, deceleration, and speed changes, demonstrating better optimization performance compared to other current control algorithms. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the control system described in this invention;

[0046] Figure 2 This is a schematic diagram of the communication mode of the vehicle fleet described in this invention;

[0047] Figure 3 This is a schematic diagram of the parameter identification method used in this invention;

[0048] Figure 4 This is a schematic diagram of the simulation scenario of the present invention; wherein (a) is the vehicle position curve during the driving process on the simulated road; (b) is the spacing curve between each vehicle during the driving process on the simulated road; (c) is the spacing error curve between the simulated road and the expected spacing during the driving process on the simulated road; (d) is the vehicle speed curve during the driving process on the simulated road; and (e) is the vehicle acceleration curve during the driving process on the simulated road. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Step 1: Establish a third-order dynamic model that can be used to identify the dynamic model parameters of heterogeneous vehicles:

[0051]

[0052] Since the actual dynamic model is unknown, and considering the heterogeneity of the fleet, the vehicle dynamic parameters need to be obtained, we built a third-order dynamic model to identify this important vehicle dynamic parameter.

[0053] Where, p n，t v n，t a n，t , These represent the position, velocity, acceleration, and control input of vehicle n at time t, respectively, where Δt is the sampling interval, and τ is the input. n，t and φ n，t Let be the transmission system time parameters and input delay of vehicle n at time t. This indicates the control input after eliminating input delay.

[0054] Furthermore, using the CTH spacing strategy, the following error dynamic characteristics of vehicle n can be modeled as follows:

[0055]

[0056] Among them, e n，p，t and e n，v，t e n，a，t Let D be the distance error, speed error, and acceleration error of vehicle n relative to vehicle n-1 at time t. n，t Let be the total unknown disturbance in the following error dynamics model of vehicle n at time t, and its expression is as follows:

[0057]

[0058] After obtaining the following error dynamic model for a single vehicle, the platoon dynamic model becomes the following error dynamic model for all following vehicles in the platoon. Furthermore, the transmission system time constant... Input delay φ n，t and external disturbances D n，t These are the parameters of the vehicle dynamics model, D n，tUnable to obtain, this can be overcome by leveraging the generalization ability of reinforcement learning models.

[0059] Step 2: When the convoy is moving, the vehicles collect their own motion state data through sensors. Based on the collected motion state, the least squares method is used to identify the transmission system time parameter τ in real time online. The specific operation is as follows:

[0060] To make the reinforcement learning formation control model applicable to different heterogeneous vehicle fleets, it is proposed to use a cache memory approach to uniformly set the input delay of all vehicles to the maximum input delay φ of all vehicles in the fleet. m .

[0061] Since the time parameters of the transmission system are time-varying, we propose to use the least squares method with an adaptive forgetting factor to identify the transmission system time parameter τ in the dynamic model of step 1 in real time online. n，t Here, we first present the Laplace transform transfer function G(s) from the control input to the acceleration in the dynamic model as follows:

[0062]

[0063] Where s is the complex variable after the Laplace transform.

[0064] Considering that the least squares method with adaptive forgetting factor is not suitable for parameter identification of time-delay systems, we propose to use the t-φ-th method. m The control input at time t and the acceleration at time t are taken as an input-output data pair and subjected to a new Laplace transform. Thus, the transfer function to be identified is transformed into:

[0065]

[0066] The parametric equations are to be obtained using the first-order backward difference method, as follows:

[0067] y n，k =φ n，k θ n，k

[0068] Among them, y n，k =a n，k ,

[0069] Then, the identification parameters are to be obtained using the following recursive formula.

[0070]

[0071] Among them, K k P represents the intermediate parameter at time t. k-1 λ represents the intermediate parameter at time t-1. k-1Let t represent the forgetting factor at time t-1, and k represent the time k.

[0072] To further improve the accuracy of parameter identification, the forgetting factor λ k Adaptive adjustment is achieved through the following formula:

[0073]

[0074] In the above formula, e base , round(), and δ∈[0,1] represent the baseline error, rounding function, and sensitivity coefficient, respectively. λ min λ k The lower limit of change, ∈ k Intermediate parameters

[0075] Step 3: Design a following controller for the following vehicle using the PPO algorithm. Details are as follows:

[0076] The state s of car n at time t n，t The proposed definition is as follows:

[0077]

[0078] in, o_delay represents an intermediate variable. n，t Indicates communication delay. Indicates the identification parameters of the adjacent preceding vehicle. τ represents the control input data within the delay time window, indicating the input delay. n，t Represents its own identification parameters

[0079] The reward function for the follower controller is:

[0080]

[0081] Where, α 1，t α 2，t α 3，t These are the time-varying weights to be designed, used to dynamically determine the relative importance of minimizing the following speed error, control input, and rate of change of acceleration. n，p，max e n，v，max , and jerk max These represent the maximum allowable values ​​for following distance error, following speed error, control input, and rate of change of acceleration, respectively. (Rate of change of acceleration jerk) n，t The proposed definition is as follows:

[0082]

[0083] in, This represents the control input of the nth vehicle at time t. This represents the parameter tau obtained through identification.

[0084] The desired acceleration of vehicle i is also limited as follows:

[0085]

[0086] Among them, a min and a max These are the lower and upper limits of permissible acceleration, respectively, to avoid violating the physical limitations of the power and braking systems. min and a max -5m / s 2 and 5m / s 2 .

[0087] Step 4: The following vehicle obtains the motion status information of the adjacent preceding vehicle through V2V communication. The motion status information includes the position p of the preceding vehicle n-1. n-1，t Speed ​​v n-1，t acceleration a n-1，t Its own transmission system time parameters Considering communication latency, the following vehicle obtains its own transmission system time parameters for identification of the preceding vehicle as follows:

[0088] Step 5: Using the information from the preceding vehicle in Step 4, the following controller from Step 3 calculates the desired acceleration. Combined with series stabilization and a first-order filter to suppress acceleration oscillations, the true acceleration 'a' is obtained. i (n), thereby controlling vehicle following and achieving the goal of stable convoy driving;

[0089] Series stability can be defined as:

[0090]

[0091] H is an adjustment parameter representing the size of the time window. If the conditions in equation (7) are met, the stability of the BEV fleet can be ensured, and the safety of the vehicles can be improved.

[0092] The formula for a first-order accelerometer filter can be expressed as:

[0093]

[0094] Where filter and filter' are filter coefficients, and bias is the bias, expressed as:

[0095] bias=|a(n+1)-a(n)|

[0096] To ensure driving safety, the distance d between adjacent vehicles is... n，t It should be kept within the following range:

[0097] d l，i ≤d n， t≤d u，i

[0098] Where, d l，i and d u，i These are the lower and upper limits of the allowed distance between adjacent vehicles, obtained according to the CTH spacing policy using the following formula:

[0099]

[0100] Among them, L n-1 Let r be the length of the vehicle in front. l，n and r u，n The parameters h represent the upper and lower limits of the control spacing range, respectively. n，g The parameters are set based on the spacing strategy in the spacing error modeling.

[0101] Step 6: Repeat steps 4 to 5 continuously during the driving process to ensure that each vehicle in the convoy can maintain stable driving and achieve adaptive control optimization for heterogeneous convoys.

[0102] Figure 1 This is a flowchart of the control system described in this invention; Figure 2 This is a schematic diagram of the communication mode of the convoy described in this invention. As can be seen from the diagram, the second vehicle can collect the movement status of the lead vehicle through sensors, and each following vehicle also communicates with the adjacent leading vehicle in the same way. Figure 3 This is a schematic diagram of the parameter identification method used in this invention.

[0103] Example:

[0104] Figure 4 This is a schematic diagram of the simulation scenario. In one simulation test, a heterogeneous car platoon with one lead car and four follower cars was used; the sensor observation delay O_delay was set to 0.01s, and the control input delay was set to 0.01s. The parameters can be initialized using the following formulas: s = [20,11,2,-7,-16]m, v = [2,2,2,2,2]m / s, a = [0,0,0,0,0]m / s², tau =

[0105] [0.4,0.35,0.25,0.3,0.35], Δt=0.01s.

[0106] In this embodiment, the target convoy is controlled using the methods described in steps 1 to 6. When the convoy controlled by the proposed heterogeneous convoy formation control method is in motion, the relationship between the distance between vehicles, speed, acceleration, and position is as follows: Figure 4 As shown. Figure 4 As shown in (d), the vehicle's acceleration always varies within the range of the lead car's acceleration and does not amplify upstream in the convoy, thus ensuring passenger comfort and chordal stability. The lead car's control input varies within a specified range, which naturally limits the lead car's acceleration, such as... Figure 4 As shown in (e). The spacing between each vehicle and the one preceding it is always maintained at [d]. l,i d u,i Therefore, driving safety can be achieved, such as... Figure 4 As shown in (b). Therefore, the proposed heterogeneous fleet formation control method can achieve both driving safety and passenger comfort while keeping the fleet within the reference speed range. Furthermore, without violating input saturation limits, the string stability of the fleet can be ensured.

[0107] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A deep reinforcement learning platoon control method considering vehicle heterogeneity, characterized in that, The method comprises the following steps: Step 1, a third-order dynamic model that can be used to identify the dynamic model parameters of a heterogeneous vehicle is established: Step 2, when the vehicle fleet is moving, the vehicle collects its own motion state data through sensors, and the transmission system time parameter tau is identified in real time online by using the least square method according to the collected motion state; Step 3, a following controller of the following vehicle is designed by using the PPO algorithm; Step 4, the following vehicle obtains the motion state information of the adjacent front vehicle through V2V communication, and the motion state information comprises the position, speed, acceleration of the front vehicle and the identified transmission system time parameter of the front vehicle, and considering the communication delay, the transmission system time parameter of the front vehicle obtained by the following vehicle has a time delay; Step 5, the following controller of step 3 obtains the expected acceleration by using the motion state information of the adjacent front vehicle of step 4, and the real acceleration is obtained by combining the string stability and the first-order filter to suppress the acceleration shock, so as to control the vehicle to follow the vehicle and realize the stable driving target of the vehicle fleet; Step 6, steps 4 to 5 are repeatedly performed during driving, so that each vehicle in the vehicle fleet can maintain stable driving and realize adaptive control optimization of the heterogeneous vehicle fleet. 2.The platoon control method of deep reinforcement learning considering vehicle heterogeneity according to claim 1, wherein, Step 1 is implemented as follows: Since the real dynamic model is unknown, the vehicle needs to obtain the vehicle dynamic parameters due to the heterogeneity of the vehicle fleet, so a third-order dynamic model is established to identify the important vehicle dynamic parameters; where p n,t , v n,t , a n,t , are the position, velocity, acceleration and control input of vehicle n at time t, respectively, Δt is the sampling interval, τ n,t and φ n,t are the transmission time parameter and input delay of vehicle n at time t, respectively, and denotes the control input after removing the input delay. The following error dynamics of the vehicle n is modeled by using the CTH spacing strategy as follows: where e n,p,t and e n,v,t , e n,a,t are the vehicle n's distance error, speed error, and acceleration error relative to vehicle n-1 at time t, D n,t is the total unknown disturbance of the vehicle n's following error dynamics model, which is expressed as follows: After getting the single vehicle following error dynamics model, the platoon dynamics model is the following error dynamics model of all the following vehicles in the platoon, the transmission system time constant Input time delay φ n,t and external disturbance D n,t is the parameter of the platoon dynamics model, D n,t cannot be obtained, which is overcome by the generalization ability of the reinforcement learning model. 3.The method of claim 2, wherein, Step 2 is implemented as follows: 2-1. The input time delay of all vehicles is uniformly set to the maximum input time delay φ of all vehicles in the vehicle fleet in the manner of using cache memory m ; 2-2. Real-time online identification of time-varying transmission system time parameter τ in dynamic model using least square method with forgetting factor n,t The transfer function G(s) from control input to acceleration in Laplace transform in dynamic model is given as follows: Where s is the complex variable after Laplace transform; 2-3. Using the t-φ m The control input at time t and the acceleration at time t are taken as an input-output data pair and subjected to a new Laplace transform. Thus, the transfer function to be identified is transformed into: The first-order backward difference method is used to obtain the parameter equation as follows: y n,k = φ n,k θ n,k wherein y n,k = a n,k , Then the identification parameters are obtained by the following recursive formula where K k denotes the intermediate parameter at time t, P k-1 denotes the intermediate parameter at time t-1, λ k-1 denotes the forgetting factor at time t-1, and k denotes the kth time; Forgetting factor λ k Adaptively adjusted by the following equation: where e base , round() and δ ∈ [0, 1] are the reference error, the rounding function and the sensitivity coefficient, respectively; λ min represents the change lower limit of λ k , ∈ k is the intermediate parameter. 4.The method of claim 3, wherein, Step 3 is implemented as follows: State s of the following vehicle n at time t n,t Is defined as follows: wherein, denotes an intermediate variable, o_delay n,t denotes a communication delay, denotes a recognition parameter of the adjacent preceding vehicle, denotes control input data within a delay time window of the input time delay, τ n,t denotes a recognition parameter of the ego vehicle; The reward function of the following controller is: wherein α 1,t ,α 2,t ,α 3,t are time-varying weights to be designed to dynamically determine the relative importance of minimizing the following: the following distance error, the following speed error, the control input, and the acceleration rate of change; e n,p,max ,e n,v,max , and jerk max are the maximum values allowed for the following distance error, the following speed error, the control input, and the acceleration rate of change, respectively; and jerk n,t is defined as follows: wherein, denotes the control input of vehicle n at time t, denotes the identified parameters; Meanwhile, the expected acceleration of the vehicle n is limited as follows: where a min and a max are lower and upper limits of the allowed acceleration, respectively, to avoid violating physical limits of the power and braking systems. 5.The platoon control method of deep reinforcement learning considering vehicle heterogeneity according to claim 4, wherein, Step 4 is implemented as follows: The motion state information includes the position p n-1,t , the speed v n-1,t , the acceleration a n-1,t , and the recognized self-transmission system time parameter of the preceding vehicle n-1 Considering the communication delay, the following vehicle obtains the recognized self-transmission system time parameter of the preceding vehicle as 6.The method of claim 4, wherein, Step 5 is implemented as follows: The string stability equation is defined as: Where H is an adjustment parameter, representing the size of the time window, and if the condition in the equation is met, the stability of the BEV vehicle fleet can be ensured; The formula of the first-order acceleration filter is: Where filter and filter' are filter coefficients, and bias is a bias, which is represented as: bias = |a(n+1)-a(n)| In order to ensure the safety of driving, the distance d between adjacent vehicles n,t should be maintained in the following range: d l,i ≤d n,t ≤d u,i where d l,i and d u,i are the lower and upper bounds of the range of allowable inter-vehicle distances, obtained according to the CTH spacing policy by the following equations: where L n-1 is the vehicle length of the preceding vehicle, r l,n and r u,n are parameters that represent the upper and lower limits of the control range of the inter-vehicle distance, h n,g is a parameter set in the inter-vehicle distance strategy modeling.

Images