Vehicle queue cooperative control method fusing deep reinforcement learning communication time delay compensation

Abstract

The invention relates to a vehicle queue cooperative control method fusing deep reinforcement learning communication time delay compensation, and belongs to the technical field of intelligent vehicle control. For the problems of control error accumulation and queue instability caused by communication time delay in the vehicle-road cloud cooperative system, a solution integrating deep reinforcement learning prediction and preset performance control is provided. According to the technical scheme, the method comprises the following steps: constructing a vehicle state prediction model based on a DDPG algorithm, and generating a time delay compensation amount in real time; establishing a vehicle dynamics model containing communication time delay, and designing a sliding mode controller based on a barrier function; and dynamically adjusting the control constraint in combination with an adaptive error boundary mechanism. According to the method, through closed-loop fusion of prediction compensation and dynamic constraint, under the scene of burst communication delay (100-400 ms), the queue spacing error is reduced by 62%, the speed tracking error is still kept to be smaller than 0.3 m / s, and the queue stability and the anti-jamming capability are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent vehicle control, and relates to a vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning. Background Art

[0002] In a vehicle-road-cloud cooperative system, the information interaction between vehicles, roadside facilities, and cloud platforms is affected by wireless communication delays, resulting in a lag in the issuance of control instructions, an accumulation of vehicle following errors, and thus the problem of platoon instability. Traditional control methods often ignore the dynamic changes in communication delays, assuming that the delays are fixed or predictable, so that in practical applications, the delay changes caused by network environment fluctuations cannot be accurately compensated, reducing the overall response speed and safety of the system.

[0003] Although existing time series prediction models such as LSTM can be used to model the trend of communication delays to a certain extent, there are still problems such as weak generalization ability and large error fluctuations in the face of random and sudden, non-stationary communication environments in vehicle-road-cloud cooperative systems. At the same time, although the sliding mode controller has the advantage of strong robustness, it has defects such as uncontrollable convergence time and high-frequency chattering. Especially when facing complex traffic disturbances, the control accuracy is difficult to meet the requirements. In addition, most existing solutions decouple the prediction and control links and fail to achieve the deep closed-loop integration of "prediction-compensation-control", restricting the response speed and adaptive ability of the system.

[0004] In view of the above problems, the present invention proposes a vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning, and innovatively introduces an adaptive error boundary expansion mechanism, which can dynamically adjust the controller convergence boundary in the case of sudden disturbances or delay surges to enhance the system's tolerance and recovery ability to disturbances. This method uses deep reinforcement learning algorithms such as DDPG to learn and predict the communication delay pattern in real time, and directly embeds the prediction result into the PPC controller to construct a closed-loop coupling architecture to achieve prediction-driven dynamic control adjustment. This method breaks through the technical limitations of idealized delay assumptions and decoupling of prediction and control in traditional control strategies, provides a highly reliable and strongly adaptive cooperative control solution for large-scale connected vehicle fleets, significantly improves the safety, energy efficiency, and traffic efficiency of intelligent transportation systems, and has broad commercial application prospects. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] A vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning, comprising the following steps:

[0008] S1: Construct a vehicle driving prediction model based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Through deep reinforcement learning training with the historical driving data of autonomous vehicles, construct a vehicle driving prediction model that can predict future trajectories.

[0009] S2: Establish a vehicle dynamics model in combination with the non-linear dynamics characteristics of the vehicle, and construct a preset performance queue control framework including communication delay.

[0010] S3: By real-time monitoring the communication delay parameters between vehicles, use the trained DDPG prediction model to dynamically generate vehicle state compensation amounts according to the communication delay.

[0011] S4: Adopt a preset performance constraint strategy based on the obstacle function, combine the sliding mode control technology to implement longitudinal dynamics cooperative control for the queue vehicles, and at the same time introduce a dynamic error boundary expansion mechanism to handle dynamic interference, ensuring that the vehicle queue realizes efficient and stable driving on the preset trajectory.

[0012] Furthermore, the specific steps of S1 are as follows:

[0013] S11: Define the Actor-Critic network architecture of the DDPG algorithm.

[0014] The input layer of the Actor network is the vehicle state vector s = (v, x, a), T , where v is the vehicle speed, x is the vehicle position, a is the acceleration, and the output layer is the predicted quantity s pred . The input of the Critic network is the state s and the action a, and the output is the Q-value evaluation.

[0015] S12: Collect the historical driving data of autonomous vehicles, construct a training data set through timestamp alignment and sliding window sampling, and perform normalization processing on the data. The normalization formula is:

[0016]

[0017] where μ x is the data mean, and σ x is the standard deviation.

[0018] S13: Define the reward function:

[0019] Define the prediction accuracy reward term to punish the deviation of the predicted state from the true value.

[0020]

[0021] Add the motion smoothness constraint term to suppress the violent fluctuation of the acceleration.

[0022]

[0023] The combined total reward function is as follows:

[0024] R total = R pred + R smooth (4)

[0025] where ω1:ω2 = 7:3;

[0026] S14: During the model training process, the gradient descent method is used to minimize the error of Q-value prediction. By calculating the gradient of the Q-value, the network parameters are adjusted. The gradient descent formula is:

[0027]

[0028] S15: Update the network parameters through time difference backpropagation. The target network adopts a soft update strategy:

[0029] θ'←ζθ+(1 - ζ)θ' (6)

[0030] ω'←ζω+(1 - ζ)ω' (7)

[0031] In the above two formulas, τ is the network parameter update rate;

[0032] S15: The termination condition for training the DDPG prediction model is to select a fixed number of training times, and the fixed number of training times is denoted as flag; in each training episode, set the episode termination condition, and the set episode termination condition is: the absolute values of the position error and the speed error after normalization reach the threshold;

[0033] Furthermore, the threshold is set to 0.1.

[0034] Furthermore, the specific steps of S2 are as follows:

[0035] S21: Establish a vehicle nonlinear dynamics model, which is specifically expressed as follows:

[0036]

[0037] where, v i , x i respectively represent the position and speed of vehicle i at a certain moment, u i represents the control input of vehicle i, m i represents the mass of vehicle i, ρ represents the density of air, C A represents the comprehensive value of the air resistance coefficient, A i represents the cross-sectional area of vehicle i, g represents gravity, and f represents the rolling resistance coefficient of the tire; for simplicity, we represent the nonlinear dynamics of the vehicle as fi (m i ,v i ); Rewrite the above formula as:

[0038]

[0039] where

[0040] S22: Build a multi-vehicle collaborative scenario in the Carla simulation platform and embed a communication delay simulation module;

[0041] Adopt a feedforward communication topology (Predecessor-Following, PF), stipulate that each vehicle in the vehicle queue only receives the state information of the vehicle in front, and form a chain communication structure; the adjacency matrix A of the communication topology is defined as:

[0042]

[0043] The corresponding Laplacian matrix L satisfies:

[0044] L = diag(∑a ij ) - A(11)

[0045] Furthermore, the specific steps of S3 are as follows:

[0046] Real-time monitor the communication delay parameter Δτ between vehicles, and generate a predicted value of the state of the vehicle in front through the DDPG prediction model

[0047] Furthermore, the specific steps of S4 are as follows:

[0048] S41: Design of a prediction / actual error constraint system;

[0049] Define the consistency actual position error e 1i = x i - x lead - d desired and the consistency actual speed error e 2i = v i - v lead , where d desired is the desired safety distance, x lead , v lead is the actual state of the vehicle in front;

[0050] Define the consistency predicted position error and the consistency predicted speed error where d desired is the desired safety distance, is the predicted state of the vehicle in front;

[0051] Require the queue error e1i , e 2i , Converge to the safety boundary Δ within the preset time 1i , Δ 2i ;

[0052] Δ 1i =(ρ x∞ - ρ x∞ )e -t + ρ x∞ (12)

[0053] Δ 2i =(ρ v∞ - ρ v∞ )e -t + ρ v∞ (13)

[0054] where formula (12) is the position error boundary, formula (13) is the velocity error boundary, and the constraint conditions are |e 1i | < Δ 1i , |e 2i | < Δ 2i ,

[0055] S42: Adaptive boundary expansion mechanism;

[0056] Construct real-time monitoring variables as follows:

[0057]

[0058] When Δe 1i > 0 or Δe 2i > 0, it means that the current actual error or predicted error has exceeded the preset boundary, and the relaxation mechanism needs to be triggered;

[0059] Set the time window If the error continues to exceed the boundary within this window, it is determined that the boundary needs to be expanded;

[0060] The new boundary is composed of the superposition of the initial exponential decay term and the integral relaxation term as follows:

[0061]

[0062] When the error exceeds the boundary, the boundary is gradually relaxed through integral accumulation to avoid sudden changes in the control amount caused by hard constraints;

[0063] The gains η 1i , η 2i need to satisfy to ensure that the boundary adjustment rate matches the preset convergence rate;

[0064] S43: Design of Prescribed-Time Disturbance Observer (PTDO);

[0065] Using the PTDO, the estimation error converges to zero within the prescribed time; within.

[0066]

[0067] where L > 0 is the observer gain;

[0068] Through the Lyapunov function It is proved that:

[0069]

[0070] Ensure that the disturbance estimation error converges to zero within the preset time where S44: Sliding surface construction and error constraint mapping;

[0071] Error normalization: Map the position and velocity tracking errors to the normalized interval;

[0072]

[0073] where Δ 1i , Δ 2i are the preset error constraint boundaries;

[0074] The Barrier Function-based Sliding Mode (BFSM) surface is designed as follows:

[0075]

[0076] where p and q are power functions, satisfying p < q, to ensure a smooth transition of the sliding surface within the constraint boundaries;

[0077] S45: Design of stage-based control law;

[0078] Decompose the control input into the equivalent control μ 1i and the reaching law control μ 2i ;

[0079] The equivalent control term is designed as:

[0080]

[0081] where Γ 2i contains the disturbance estimation and the error dynamic coupling term, used to cancel the nonlinear disturbance;

[0082] Reaching law control term design, designing a reaching law to drive the system state to the sliding surface;

[0083]

[0084] where γ i > 0 is the robust gain, ensuring that the sliding surface is captured within the preset time ;

[0085] The total control law is as follows:

[0086] u i = μ 1i + μ 2i (27)

[0087] The beneficial effects of the present invention are as follows:

[0088] (1) The prediction model constructed by the Deep Deterministic Policy Gradient (DDPG) algorithm can learn the communication delay pattern in complex traffic scenarios in real time, accurately predict the state of the vehicle ahead and generate a compensation amount, effectively solving the problem of error accumulation caused by time delay in traditional methods. Experimental data shows that in the scenario of sudden communication delay (100 - 400 ms), this method reduces the queue spacing error by 62%, significantly improving the stability of multi-vehicle cooperation.

[0089] (2) By embedding the DDPG prediction compensation mechanism and integrating the Prescribed Time Disturbance Observer (PTDO) with the new sliding surface (BFSM), the error convergence efficiency of the proposed method within the preset time is significantly improved compared with the traditional control scheme. Moreover, relying on the dynamic prediction and compensation ability of DDPG for time-varying communication delay, the chattering problem commonly existing in traditional sliding mode control is effectively suppressed, and the control performance achieves an important breakthrough compared with the traditional method that simply relies on PTDO and BFSM.

[0090] (3) Innovatively combining the obstacle function with sliding mode control, designing a Preset Performance Controller (PPC) with a dynamic error boundary expansion mechanism, while ensuring the strict convergence of the tracking error, allowing the boundary to adaptively expand according to the disturbance intensity, greatly enhancing the anti-interference ability in complex traffic environments.

[0091] (4) Constructing a deep coupling framework of the DDPG prediction model and the PPC controller, realizing a closed-loop of prediction compensation and control response at the millisecond level (100 - 400 ms). Simulation experiment data shows that this architecture enables the vehicle queue to maintain a speed tracking error of less than 0.3 m / s in the scenario of sudden communication delay (100 - 400 ms).

[0092] Other advantages, objects, and features of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the means of the instrumentalities and combinations particularly pointed out hereinafter. Brief Description of the Drawings

[0093] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, where:

[0094] Figure 1 is a flowchart of the vehicle platoon cooperative control method integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention;

[0095] Figure 2 is a structural diagram of the DDPG prediction model neural network controller in an embodiment of the present invention;

[0096] Figure 3 is an architecture diagram of the vehicle platoon cooperative control method integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention;

[0097] Figure 4 is a speed prediction result diagram of the DDPG algorithm in an embodiment of the present invention;

[0098] Figure 5 is an acceleration prediction result diagram of the DDPG algorithm in an embodiment of the present invention;

[0099] Figure 6 is a position prediction result diagram of the DDPG algorithm in an embodiment of the present invention;

[0100] Figure 7 is a position result diagram of the vehicle platoon cooperative control integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention;

[0101] Figure 8 is a speed result diagram of the vehicle platoon cooperative control integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention;

[0102] Figure 9 is a consistency speed error result diagram of the vehicle platoon cooperative control integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention;

[0103] Figure 10 is a consistency position error result diagram of the vehicle platoon cooperative control integrating deep reinforcement learning communication delay compensation in an embodiment of the present invention. Detailed Embodiments

[0104] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0105] Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0106] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and should not be construed as a limitation to the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0107] The present invention provides a vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning to improve the platoon consistency of autonomous vehicles in complex traffic environments. The method includes the following steps:

[0108] S1: Construct a vehicle driving prediction model based on the Deep Deterministic Policy Gradient (DDPG) algorithm, and perform deep reinforcement learning training through the historical driving data of autonomous vehicles to construct a vehicle driving prediction model that can predict future trajectories;

[0109] S2: Establish a vehicle dynamics model in combination with the nonlinear dynamics characteristics of the vehicle, and construct a preset performance queue control framework including communication delay;

[0110] S3: Dynamically generate a vehicle state compensation amount according to the communication delay by using the trained DDPG prediction model through real-time monitoring of the communication delay parameters between vehicles;

[0111] S4: Using a preset performance constraint strategy based on an obstacle function, combined with sliding mode control technology, the longitudinal dynamics cooperative control of the platoon vehicles is implemented. At the same time, a dynamic error boundary expansion mechanism is introduced to handle dynamic interference, ensuring that the vehicle platoon can achieve efficient and stable driving on the preset trajectory; the implementation of the vehicle platoon cooperative control integrating deep reinforcement learning and communication delay compensation using the method of the present invention is as follows: Figures 1 to 3 shown.

[0112] Furthermore, S1 specifically includes the following steps:

[0113] S11: Build the Actor prediction network, the input layer receives the vehicle state vector s = (v, x, a) T (historical speed, position, acceleration), the output layer generates the predicted state within the future ΔT time window

[0114] S12: Design the Critic evaluation network, the input is the current state s t and Actor predicted actions Output Q value function Q(s,a), which is used to quantify the deviation between the predicted state and the actual state;

[0115] S13: Collect historical driving data of autonomous vehicles and perform timestamp alignment and outlier cleaning on the raw data;

[0116] First, a training data set is constructed by timestamp alignment and sliding window sampling, and the data is normalized. The normalization formula is:

[0117]

[0118] Among them, μ x is the data mean, σ x is the standard deviation;

[0119] Then, based on the 3σ criterion, the acceleration jump points are eliminated and the interpolation formula is a corrected =median(a t-2 ,a t-1 ,a t+1 ,a t+2 );

[0120] Finally, the data enhancement strategy is implemented to add Gaussian noise s'=s+∈·Ν(0,σ 2 ), where ∈~Bernoulli (p=0.3) is random noise;

[0121] S14: Design a multi-objective optimization reward function;

[0122] Define a prediction accuracy reward item to penalize the deviation of the predicted state from the true value;

[0123]

[0124] Add a motion smoothness constraint term; suppress sharp fluctuations in acceleration;

[0125]

[0126] Finally, the synthesized total reward function is:

[0127] R total = R pred + R smooth (4)

[0128] where ω1:ω2 = 7:3;

[0129] S15: Model training and convergence optimization;

[0130] Adopt a prioritized experience replay mechanism and define the sample priority p i = |δ i | + ∈, where δ i = Q(s,a|θ Q ), y i is the temporal difference error, y i = R i + γQ'(s', μ'(s')|θ Q ') is the target Q value;

[0131] During the model training process, use the gradient descent method to minimize the error of Q value prediction, and adjust the network parameters by calculating the gradient of the Q value; the gradient descent formula is:

[0132]

[0133] Design an exploration noise decay strategy, and the variance decay formula of the action noise is:

[0134] σ t = σ0·e -kt (6)

[0135] In the above formula, the initial value σ0 = 0.3 and the decay coefficient k = 0.95;

[0136] Implement a double Q-network decay strategy, update the network parameters through the backpropagation of the temporal difference (TD Error), and the target network adopts a soft update strategy. The update formula is:

[0137] θ'←ζθ+(1 - ζ)θ' (7)

[0138] ω'←ζω+(1 - ζ)ω' (8)

[0139] In the above two equations, τ is the network parameter update rate;

[0140] Finally, it is the termination condition for training the DDPG prediction model. In this method, reaching a fixed number of training times is selected, and reaching the fixed number of training times is denoted as flag; in each training episode, it is also necessary to set the episode termination condition. The set episode termination condition is that the absolute value of the normalized position error and speed error reaches a threshold, and the threshold is set to 0.1;

[0141] Furthermore, S2 specifically includes the following steps:

[0142] S21: Establish a vehicle nonlinear dynamics model, which is specifically expressed as follows:

[0143]

[0144] Among them, v i , x i respectively represent the position and speed of vehicle i at a certain moment, u i represents the control input of vehicle i, m i represents the mass of vehicle i, ρ represents the density of air, C A represents the comprehensive value of the air resistance coefficient, A i represents the cross-sectional area of vehicle i, g represents gravity, and f represents the rolling resistance coefficient of the tire. For simplicity, we represent the nonlinear dynamics of the vehicle as f i (m i , v i ), and thus rewrite the above equation as:

[0145]

[0146] Among them,

[0147] S22: Build a multi-vehicle collaborative scenario in the Carla simulation platform;

[0148] Build a digital twin road network of urban roads in the Carla engine, and define road curvature, slope, lane line attributes, and traffic sign semantic information through the OpenDRIVE standard; at the same time, configure multiple controlled experimental vehicles (CAVs) with cooperative driving functions. Each vehicle loads the nonlinear dynamics model parameters described in S21, and realizes real-time interaction of vehicle control commands through the Carla Python API;

[0149] S23: Build a ROS2-based distributed communication middleware at the simulation node layer, and define the Topic subscription / publishing mechanism for vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I); design a custom message format to encapsulate vehicle status data packets, including fields such as location, speed, acceleration, control instructions, and timestamps, and the data packet structure adopts the Protocol Buffers serialization protocol;

[0150] S24: Develop a delay injection middleware and insert programmable delay nodes into the data packet transmission path;

[0151] S25: Adopt a feed-forward communication topology (Predecessor-Following, PF), stipulate that each vehicle in the vehicle queue only receives the status information of the vehicle in front, and form a chain communication structure. The adjacency matrix A of the communication topology is defined as:

[0152]

[0153] The corresponding Laplacian matrix L satisfies:

[0154] L = diag(∑a ij ) - A(12)

[0155] Furthermore, S3 specifically includes the following steps:

[0156] S31: Deploy a delay detection unit in the vehicle network system to monitor the round-trip time (RTT) of vehicle-to-vehicle and vehicle-to-infrastructure communications in real time;

[0157] S32: Align the time stamps of the delay detection results with the vehicle status information (location, speed, acceleration, etc.) to prepare for subsequent model input;

[0158] S33: Normalize the delay data and its associated vehicle status data to ensure that data with different dimensions can be uniformly input into the DDPG prediction model;

[0159] S34: Load and initialize the model;

[0160] Load the pre-offline trained DDPG model (including the Actor network and the Critic network) in the control system; initialize the network weights to the latest or optimal version to ensure that the model has good prediction and decision-making capabilities in the current traffic environment;

[0161] S35: Dynamically generate the delay compensation amount;

[0162] Real-time monitor the vehicle-to-vehicle communication delay parameter Δτ, and generate the predicted value of the status of the vehicle in front through the DDPG prediction model Set a safety threshold or upper and lower limits to prevent the predicted value from being too large or too small due to model errors or abnormal inputs, thereby avoiding vehicle control instability or safety hazards;

[0163] Further, S4 specifically includes the following steps:

[0164] S41: Design of the prediction / actual error constraint system;

[0165] Define the consistency actual position error e 1i = x i - x lead - d desired and the consistency actual speed error e 2i = v i - v lead , where d desired is the desired safety distance, x lead , v lead is the actual state of the vehicle in front;

[0166] Define the consistency predicted position error and the consistency predicted speed error where d desired is the desired safety distance, is the predicted state of the vehicle in front;

[0167] Require that the queue errors e 1i , e 2i , converge to the safety boundary Δ within the preset time 1i , Δ 2i ;

[0168] Δ 1i = (ρ x∞ - ρ x∞ ) e -t + ρ x∞ (13)

[0169] Δ 2i = (ρ v∞ - ρ v∞ ) e -t + ρ v∞ (14)

[0170] where formula (13) is the position error boundary and formula (14) is the speed error boundary, and the constraint conditions are |e 1i | < Δ 1i , |e 2i | < Δ 2i ,

[0171] S42: Adaptive boundary expansion mechanism;

[0172] Construct real-time monitoring variables as follows:

[0173]

[0174] When Δe 1i > 0 or Δe 2i > 0, it indicates that the current actual error or predicted error has exceeded the preset boundary, and the relaxation mechanism needs to be triggered;

[0175] Set the time window If the error continues to exceed the boundary within this window, it is determined that the boundary needs to be expanded;

[0176] The new boundary is composed of the superposition of the initial exponential decay term and the integral relaxation term; as follows:

[0177]

[0178] When the error exceeds the boundary, the boundary is gradually relaxed through integral accumulation to avoid sudden changes in the control quantity caused by hard constraints;

[0179] Gain η 1i , η 2i Needs to satisfy Ensure that the boundary adjustment rate matches the preset convergence rate;

[0180] S42: Design of the Prescribed-Time Disturbance Observer (PTDO);

[0181] Using the PTDO, the estimation error can converge to zero within the prescribed time ;

[0182]

[0183] where L > 0 is the observer gain;

[0184] Through the Lyapunov function It is proved that:

[0185]

[0186] Ensure that the disturbance estimation error converges to zero within the preset time , where S43: Construction of the Sliding Mode Surface (BFSM) and Error Constraint Mapping;

[0187] Perform error normalization processing to map the position and velocity tracking errors to the normalized interval;

[0188]

[0189] In the above formula, Δ 1i , Δ2i is the preset error constraint boundary;

[0190] The barrier function-based sliding mode (BFSM) surface is designed as:

[0191]

[0192] where p and q are power functions, satisfying p < q to ensure smooth transition of the sliding mode surface within the constraint boundary; S44: Design of the stage control law;

[0193] The control input is decomposed into the equivalent control μ 1i and the reaching law control μ 2i ;

[0194] The equivalent control term is designed as:

[0195]

[0196] where Γ 2i contains the disturbance estimation and the error dynamic coupling term, used to cancel the nonlinear disturbance; Design of the reaching law control term, designing the reaching law to drive the system state to the sliding mode surface;

[0197]

[0198] where γ i > 0 is the robust gain to ensure that the sliding mode surface is captured within the preset time ;

[0199] The total control law is:

[0200] u i = μ 1i + μ 2i (28)

[0201] Figure 4 This is the speed prediction result graph of the DDPG algorithm in the example of the present invention; Figure 5 This is the acceleration prediction result graph of the DDPG algorithm in the example of the present invention; Figure 6 This is the position prediction result graph of the DDPG algorithm in the example of the present invention; Figure 7 This is the position result graph of the vehicle platoon cooperative control with communication delay compensation by integrating deep reinforcement learning in the example of the present invention; Figure 8 This is the speed result graph of the vehicle platoon cooperative control with communication delay compensation by integrating deep reinforcement learning in the example of the present invention; Figure 9 This is the consistency speed error result graph of the vehicle platoon cooperative control with communication delay compensation by integrating deep reinforcement learning in the example of the present invention; Figure 10 This is the consistency position error result graph of the vehicle platoon cooperative control with communication delay compensation by integrating deep reinforcement learning in the example of the present invention.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the present technical solution, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning, characterized in that: It includes the following steps: S1: Construct a vehicle driving prediction model based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Through deep reinforcement learning training with the historical driving data of autonomous vehicles, construct a vehicle driving prediction model that can predict future trajectories. S2: Establish a vehicle dynamics model in combination with the non-linear dynamics characteristics of the vehicle, and construct a preset performance queue control framework including communication delay. S3: By real-time monitoring the communication delay parameters between vehicles, use the trained DDPG prediction model to dynamically generate vehicle state compensation amounts according to the communication delay. S4: Adopt a preset performance constraint strategy based on the barrier function, and combine the sliding mode control technology to implement longitudinal dynamics cooperative control for the queue vehicles. At the same time, introduce a dynamic error boundary expansion mechanism to handle dynamic interference, ensuring that the vehicle queue can drive efficiently and stably on the preset trajectory.

2. The vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning according to claim 1, wherein: The specific steps of S1 are as follows: S11: Define the Actor-Critic network architecture of the DDPG algorithm. The input layer of the Actor network is the vehicle state vector s = (v, x, a) T , where v is the vehicle speed, x is the vehicle position, a is the acceleration, and the output layer is the predicted quantity s pred , the input of the Critic network is the state s and the action a, and the output is the Q-value evaluation; S12: Collect the historical driving data of autonomous vehicles, construct a training data set through timestamp alignment and sliding window sampling, and normalize the data. The normalization formula is: Among them, μ x is the data mean, and σ x is the standard deviation; S13: Define the reward function: Define the prediction accuracy reward term to punish the deviation of the predicted state from the true value. Add a motion smoothness constraint term to suppress the violent fluctuation of acceleration. Synthesize the total reward function as follows: R total = R pred + R sm ooth (4) where ω1:ω2 = 7:3; S14: During the model training process, use the gradient descent method to minimize the error of Q-value prediction. By calculating the gradient of the Q-value, adjust the network parameters. The gradient descent formula is: S15: Update the network parameters through temporal difference backpropagation. The target network adopts a soft update strategy: θ'←ζθ+(1-ζ)θ' (6) ω'←ζω+(1-ζ)ω' (7) In the above two formulas, τ is the network parameter update rate. S15: The termination condition for DDPG prediction model training is to select the achievement of a fixed number of training times, and record the achievement of the fixed number of training times as flag. In each training episode, set the episode termination condition. The set episode termination condition is that the absolute values of the position error and speed error after normalization reach the threshold.

3. A vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning, characterized in that: The threshold is set to 0.

1.

4. A vehicle platoon cooperative control method integrating communication delay compensation of deep reinforcement learning according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: Establish a vehicle non-linear dynamics model, which is specifically expressed as follows: where, v i , x i represent the position and speed of vehicle i at a certain moment respectively, u i represents the control input of vehicle i, m i represents the mass of vehicle i, ρ represents the density of air, C A represents the comprehensive value of the air resistance coefficient, A i represents the cross-sectional area of vehicle i, g represents gravity, f represents the rolling resistance coefficient of the tire; for simplicity, we represent the nonlinear dynamics of the vehicle as f i (m i , v i ); rewrite the above equation as: Among them, S22: Build a multi-vehicle cooperation scenario in the Carla simulation platform and embed a communication delay simulation module. Adopt the Predecessor-Following (PF) communication topology, and stipulate that each vehicle in the vehicle queue only receives the state information of the preceding vehicle to form a chain communication structure. The adjacency matrix A of the communication topology is defined as: The corresponding Laplacian matrix L satisfies: L = diag(∑a ij ) - A (11).

5. A vehicle queue cooperative control method integrating communication delay compensation based on deep reinforcement learning, characterized in that: The specific steps of S3 are as follows: Real-time monitor the communication delay parameter Δτ between vehicles, and generate the predicted value of the state of the vehicle ahead through the DDPG prediction model 6. A vehicle platoon cooperative control method integrating communication delay compensation based on deep reinforcement learning, characterized in that: The specific steps of S4 are as follows: S41: Predict / actual error constraint system design; Define the consistency actual position error e 1i = x i - x lead - d desired And the consistency actual speed error e 2i = v i - v lead , where d desired is the expected safety distance, x lead , v lead are the actual states of the vehicle ahead; Define the consistency prediction position error and the consistency prediction speed error where d desired is the desired safety distance, is the predicted state of the vehicle ahead; Required queue error e 1i , e 2i , within a preset time converge to the safety boundary Δ 1i , Δ 2i inside; Δ 1i = (ρ x∞ - ρ x∞ )e -t + ρ x∞ (12) Δ 2i = (ρ v∞ - ρ v∞ ) e -t + ρ v∞ (13) Among them, formula (12) is the position error boundary, formula (13) is the velocity error boundary, and the constraint condition is |e 1i | < Δ 1i , |e 2i | < Δ 2i , S42: Adaptive boundary expansion mechanism; Construct a real-time monitoring variable, as follows: When Δe 1i > 0 or Δe 2i > 0, it indicates that the current actual error or predicted error has exceeded the preset boundary, and the relaxation mechanism needs to be triggered; Set a time window If the error continues to exceed the boundary within this window, it is determined that the boundary needs to be expanded; The new boundary is composed of the superposition of the initial exponential decay term and the integral relaxation term, as follows: When the error exceeds the boundary, gradually relax the boundary through integral accumulation to avoid sudden changes in the control quantity caused by hard constraints; Gain η 1i , η 2i needs to satisfy to ensure that the boundary adjustment rate matches the preset convergence rate; S43: Design of Prescribed-Time Disturbance Observer (PTDO); Using PTDO, the estimation error converges to zero within a specified time interval; where L > 0 is the observer gain; Via Lyapunov function Proof: Ensure that the disturbance estimation error converges to zero within a preset time , where S44: Sliding surface construction and error constraint mapping; Error normalization processing: Map the position and velocity tracking errors to the normalized interval; where Δ 1i , Δ 2i are preset error constraint boundaries; The design of the Barrier Function-based Sliding Mode (BFSM) surface is as follows: where p and q are power functions, satisfying p < q to ensure smooth transition of the sliding surface within the constraint boundary; S45: Design of the phased control law; Decompose the control input into the equivalent control μ 1i and the reaching law control μ 2i ; The equivalent control term is designed as: Among them, Γ 2i includes a disturbance estimate and an error dynamic coupling term for canceling the nonlinear disturbance; Design of the reaching law control term to drive the system state to the sliding surface; where γ i > 0 is the robust gain to ensure that the sliding surface is captured within a preset time ; The total control law is as follows: u i = μ 1i + μ 2i (27).

Images