A Robust Reinforcement Learning-Based Cooperative Control Method for Electric Vehicle Formation
By introducing robust reinforcement learning and adversarial perturbation training into electric vehicle formation control, the performance degradation problem of deep reinforcement learning in complex environments is solved, and stable and efficient vehicle formation control is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2026-04-03
AI Technical Summary
Deep reinforcement learning models in electric vehicle platooning control rely too heavily on the distribution of training data, leading to performance degradation in complex traffic environments and making it difficult to promote in real-world driving scenarios.
A robust reinforcement learning mechanism is introduced to enhance the generalization ability of the model by training it in a simulated environment. Combined with Fast Gradient Sign Method (FGSM) adversarial perturbation and progressive learning, the robustness and control performance of the algorithm in complex environments are improved.
It achieves stable and reliable vehicle platooning control in complex road environments, improving the overall performance and control effect of intelligent connected vehicle fleets.
Smart Images
Figure CN118605526B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent driving, specifically relating to a cooperative control method for electric vehicle platooning based on robust reinforcement learning. Background Technology
[0002] In the control of connected vehicle fleets, cooperative platooning control of electric vehicles is a core technology, involving the ability of vehicles to automatically maintain appropriate distances and respond to changes in the speed of the vehicle ahead. Traditional control methods, such as PID control and model-based predictive control, while reliable in certain scenarios, suffer from limitations in adaptability and flexibility when facing complex traffic environments and variable road conditions. In recent years, deep reinforcement learning (DRL) has been introduced as an advanced control strategy, attracting significant attention due to its ability to learn optimal control strategies through interaction with the environment.
[0003] However, when deep reinforcement learning is applied to vehicle formation control, it often faces the problem that the model is too dependent on the distribution of training data, which leads to performance degradation in the presence of disturbances, thus limiting its promotion in real driving scenarios. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a robust reinforcement learning-based cooperative control method for intelligent connected electric vehicle platooning. This method introduces a robust training mechanism on top of Deep Reinforcement Learning (DRL) to enhance the model's generalization ability and robustness. Through training in simulated environments with different driving cycles, the model is trained to predict and adapt to changing environments, achieving more stable and reliable vehicle platooning control. This overcomes the limitations of traditional control methods and aims to improve the overall performance of intelligent connected vehicle fleets in complex road environments through deep reinforcement learning and robust training. This robust reinforcement learning control method not only optimizes the control structure but also provides an important technological foundation for the future development of intelligent transportation systems. The specific design follows these steps:
[0005] Step 1: Establish a third-order longitudinal dynamics model for vehicle i.
[0006] Step 2: Based on the third-order longitudinal dynamics model established in Step 1, select the control objective:
[0007] The actual distance d between vehicle i and the vehicle in front at time t i (t) is represented as:
[0008] d i (t)=p i-1 (t)-p i (t)-L i
[0009] Among them, Li Let p be the length of vehicle i. i (t) represents the position of vehicle i. To ensure vehicle safety, a minimum distance is set. To avoid collisions between the main vehicle and the vehicle in front; and to set a maximum distance. To prevent other vehicles from cutting into the queue. The traffic safety control objective is:
[0010]
[0011] in, v i (t) is the speed of vehicle i.
[0012] To achieve vehicle platooning spacing control, the spacing deviation is calculated:
[0013]
[0014] in, This represents the expected spacing at time t. The goal of spacing tracking is to control the difference between the actual spacing and the expected spacing. minimize.
[0015] To achieve driving comfort, the maximum acceleration value 'a' is set. max and the minimum acceleration a min , Let be the desired acceleration at time t. The target for ride comfort control is:
[0016]
[0017] To achieve cooperative control of electric vehicle platooning, the state error of vehicle i is defined as:
[0018]
[0019] Among them, a i (t) is the acceleration of vehicle i. This indicates the difference in acceleration between the main vehicle and the vehicle in front. This indicates the speed difference between the main vehicle and the vehicle in front. This indicates the speed deviation between the lead vehicle and the speed of the vehicle in front. The objective for efficient convoy following control is: and minimize.
[0020] Step 3: Based on the control objective, construct the reward function, state variables, and control variables in the robust reinforcement learning algorithm framework.
[0021] In the robust reinforcement learning algorithm framework, the input is a state variable. Including the actual distance d between vehicle i and the vehicle in front at time ti (t) and the expected distance d des Deviation between Speed deviation between vehicle i and the vehicle in front Acceleration deviation between vehicle i and the vehicle in front Speed deviation between vehicle i and the lead vehicle The output action is the control variable Con. i (t), which is the expected acceleration of vehicle i at time t.
[0022] To meet the above control objectives, the reward function Reward... i (S i (t), Con i (t) includes the reward component Re and the penalty component Pe.
[0023] Step 4: Based on state variables, control variables, and reward functions, construct adversarial samples and use the reinforcement learning algorithm PPO as the controller for the intelligent agent vehicle to complete the vehicle platooning cooperative control.
[0024] Based on the state variables, control variables, and reward function designed in step 3, the reinforcement learning algorithm PPO is used as the controller for the intelligent agent vehicle. The Fast Gradient Sign Method (FGSM) is employed to apply adversarial perturbations to the samples during training, improving the algorithm's robustness. At time t, the input state passed from the environment to agent i is S. i (t), the reward is Reward i The Critic network calculates the gradient of the current network weight parameters with respect to the input state as follows:
[0025]
[0026] Where ω represents the network weight parameters of the current Critic network, and L is the loss function of the Critic network. The perturber receives the gradient grad passed from the Critic network and calculates the perturbation using the FGSM method:
[0027] δ=∈*sign(grad)
[0028] Where ∈ is the perturbation coefficient. sign is the sign function, which determines the sign of the gradient. The perturbator adds this perturbation to the current state, resulting in the adversarial sample:
[0029] S′ i (t)=S i (t)+δ
[0030] The Actor network receives S′ transmitted by the perturber. i(t), and output the action Con according to the weight parameters of the Actor network. i (t).
[0031] Step 4: By adopting a progressive learning method, the training phase begins with a small fleet and gradually increases the number of following vehicles, thereby gradually improving the longitudinal coordination and control capabilities of the fleet formation in a multi-vehicle environment.
[0032] Compared with the prior art, the advantages of the present invention are as follows:
[0033] 1) By using the FGSM method, perturbation samples are applied during the control training process of longitudinal driving in vehicle formation, which improves the robustness of the algorithm in complex environments.
[0034] 2) By adopting a course-based learning approach and gradually increasing the fleet size, the control effect of the algorithm in multi-vehicle collaborative control can be improved. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is the control framework of the method described in this invention;
[0037] Figure 2 This is a diagram showing the spacing between each following vehicle;
[0038] Figure 3 It's a speed graph for each vehicle;
[0039] Figure 4 This is a speed deviation diagram between each vehicle and the lead vehicle;
[0040] Figure 5 It shows the acceleration graph for each vehicle;
[0041] Figure 6 It is a multi-scenario test reward graph with different numbers of fleets and stable model weights. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] A robust reinforcement learning-based cooperative control method for electric vehicle formations, with the control framework as follows: Figure 1 As shown, it includes the following steps:
[0044] Step 1: Establish the third-order longitudinal dynamics model of vehicle i as follows:
[0045]
[0046] Where p i (t), v i (t), a i (t) and These are the position, velocity, acceleration, and desired acceleration of vehicle i, respectively; τ i It is the mechanical inertia of vehicle i.
[0047] Step 2: Based on the third-order longitudinal dynamics model established in Step 1, select the control objective:
[0048] The actual distance d between vehicle i and the vehicle in front at time t i (t) is represented as:
[0049] d i (t)=p i-1 (t)-p i (t)-L i
[0050] Among them, L i Let be the length of vehicle i. To ensure vehicle safety, a minimum distance is set. To avoid collisions between the main vehicle and the vehicle in front; and to set a maximum distance. To prevent other vehicles from cutting into the queue. The traffic safety control objective is:
[0051]
[0052] in,
[0053] To achieve vehicle platooning spacing control, the spacing deviation is calculated:
[0054]
[0055] in, This represents the expected spacing at time t. The goal of spacing tracking is to control the difference between the actual spacing and the expected spacing. The smaller the size, the better for controlling vehicle spacing.
[0056] To achieve driving comfort, the maximum acceleration value 'a' is set. max =2.5m / s 2 and the minimum acceleration amin = -2.5m / s 2 The target for controlling ride comfort is:
[0057]
[0058] To achieve cooperative control of electric vehicle platooning, the state error of vehicle i is defined as:
[0059]
[0060] in, This indicates the difference in acceleration between the main vehicle and the vehicle in front. This indicates the speed difference between the main vehicle and the vehicle in front. This indicates the speed deviation between the lead vehicle and the speed of the vehicle in front. The objective for efficient convoy following control is: and Minimize. The goal of efficient platooning control is to make the overall coordinated control of vehicle formations more efficient.
[0061] Step 3: In the robust reinforcement learning algorithm framework, the input is the state variable. The output action is the control variable Con. i (t), which is the expected acceleration of vehicle i at time t.
[0062] To meet the above control objectives, the reward function Reward... i (S i (t), Con i (t) includes a reward component Re and a penalty component Pe. The reward component Re is designed as follows:
[0063] Re = α1 * R distance +α2*R flow
[0064] Where α1 and α2 are the weights of the reward component. R distance R flow The reward items corresponding to meeting the spacing control target and driving follow efficiency are defined as follows:
[0065]
[0066] Among them, k1, k2, and k3 are weight parameters.
[0067] The penalty term Pe is used to penalize situations where the spacing deviation is too small or too large during training, thereby promoting the algorithm to learn a better spacing maintenance strategy. Corresponding to the above-mentioned driving safety control objective, Pe is designed as follows:
[0068]
[0069] Therefore, the reward function is expressed as:
[0070] Reward i (S i (t), Con i (t))=Re+Pe
[0071] Step 4: Based on the state variables, control variables, and reward function designed in Step 3, the reinforcement learning algorithm PP0 is used as the controller for the intelligent vehicle. The Fast Gradient Sign Method (FGSM) is employed to apply adversarial perturbations to the samples during training, improving the algorithm's robustness. At time t, the input state passed from the environment to agent i is S. i (t), the reward is Reward i The Critic network calculates the gradient of the current network weight parameters with respect to the input state as follows:
[0072]
[0073] Where ω represents the network weight parameters of the current Critic network, and L is the loss function of the Critic network. The perturber receives the gradient grad passed from the Critic network and calculates the perturbation using the FGSM method:
[0074] δ=∈*sign(grad)
[0075] Here, ∈ is a small perturbation coefficient to ensure that the perturbation is not too large and causes the input state to become unrealistic. sign is the sign function, which determines the sign of the gradient. The perturbator adds this perturbation to the current state to obtain the adversarial sample:
[0076] S′ i (t)=S i (t)+δ
[0077] The Actor network receives S′ transmitted by the perturber. i (t), and output the action Con according to the weight parameters of the Actor network. i (t).
[0078] Step 4: By adopting a progressive learning method, this study started with a small fleet of vehicles in the training phase and gradually increased the number of following vehicles, thereby gradually improving the longitudinal collaborative control capability of the fleet formation in a multi-vehicle environment.
[0079] Figure 2 , Figure 3 , Figure 4 and Figure 5The diagrams show the spacing, speed, speed deviation, and acceleration of a convoy of four vehicles, demonstrating the algorithm's good performance in longitudinal control of vehicle formation. Figure 2 The algorithm demonstrates its performance in spacing control, with each following vehicle able to maintain its spacing within the gray area formed by the maximum and minimum spacing, thus verifying the safety control objectives proposed by the algorithm. Figure 3 It demonstrates the speed following control effect, showing that the following vehicle can maintain speed tracking even with complex speed changes. Figure 4 It shows the speed difference between each following vehicle and the lead vehicle, demonstrating that the convoy meets the requirements for platoon stability. Figure 5 The acceleration control effect is demonstrated, and the following vehicle can maintain acceleration tracking under complex acceleration changes. Table 1 shows a comparison of the control performance of the proposed Robust_PPO method and the PPO control algorithm on 10 vehicles in the UDDS driving cycle, showing improvements in spacing, speed, and acceleration control.
[0080] Example:
[0081] In this embodiment, a smart connected vehicle fleet consisting of one lead vehicle and three follower vehicles is constructed and controlled using the Robust_PPO method described in steps one through three. By modeling the vehicle environment, state variables, action variables, and reward functions are designed, such as... Figure 6 The test reward graphs for different numbers of vehicle fleets with varying weights are shown in Table 1. During training, the FGSM is used to perturb the input environment state, and a course learning method is adopted for training, thereby improving the robustness of the algorithm. The experimental comparison results are shown in Table 1.
[0082] Table 1 Comparison of control performance between the Robust_PPO algorithm and the PPO algorithm proposed in this invention.
[0083]
[0084] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.
Claims
1. A cooperative control method for electric vehicle formations based on robust reinforcement learning, characterized in that, Includes the following steps: Step 1: Create a vehicle A third-order longitudinal dynamic model; Step 2: Select the control objective based on the third-order longitudinal dynamics model; Step 3: Based on the control objective, construct the reward function, state variables, and control variables in the robust reinforcement learning algorithm framework; The specific control objectives are as follows: and Representing vehicles Position, velocity, acceleration, and desired acceleration, Time vehicle Actual distance from the vehicle in front Represented as: ； in, For vehicles The vehicle length is set to the minimum distance. To avoid collisions between the main vehicle and the vehicle in front; and to set a maximum distance. To prevent other vehicles from cutting into the queue; the traffic safety control objective is: ； in, ; To achieve vehicle platooning spacing control, the spacing deviation is calculated: ； in, express The desired distance at any given time, with the distance tracking objective being to control the difference between the actual distance and the desired distance. minimize; Set the maximum acceleration value and minimum acceleration , for The expected acceleration at any given moment; the target for driving comfort control is: ； To achieve cooperative control of electric vehicle platooning, define vehicles The state error is: ； ； ； in, This indicates the difference in acceleration between the main vehicle and the vehicle in front. This indicates the speed difference between the main vehicle and the vehicle in front. This indicates the speed deviation between the lead vehicle and the lead vehicle; the objective of efficient convoy following control is: , and minimize; The specific implementation process of step 3 is as follows: In the robust reinforcement learning algorithm framework, the input is a state variable. The output action is a control variable. That is, vehicles exist Expected acceleration at any moment ; To satisfy the control objective, the reward function Including the reward portion and punishment section ; The reward portion Designed as follows: ； in, , This refers to the weight of the reward portion; The reward items corresponding to meeting the spacing control target and driving follow efficiency are defined as follows: ； ； in, , , These are weight parameters; Penalty items Designed as follows: ； Therefore, the reward function is expressed as: ； Step 4: Based on state variables, control variables, and reward functions, construct adversarial examples and use the reinforcement learning algorithm PPO as the controller for the intelligent agent vehicle to complete vehicle platooning cooperative control. Step 5: By adopting a progressive learning method, gradually increase the number of following vehicles during the training phase to gradually improve the longitudinal coordination and control capabilities of the convoy formation in a multi-vehicle environment.
2. The electric vehicle formation cooperative control method based on robust reinforcement learning according to claim 1, characterized in that, The vehicle The third-order longitudinal dynamic model is as follows: ； in It is a vehicle The mechanical inertia.
3. The electric vehicle formation cooperative control method based on robust reinforcement learning according to claim 2, characterized in that, The specific implementation process of step 4 is as follows: Based on the state variables, control variables, and reward function designed in step 3, the reinforcement learning algorithm PPO is used as the controller for the intelligent vehicle; the Fast Gradient Sign Method (FGSM) is employed to apply adversarial perturbations to the samples during the training process. At any moment, the environment transmits information to the intelligent agent. The input state is The reward is The Critic network calculates the gradient of the current network weight parameters with respect to the input state as follows: ； in, This represents the network weight parameters of the current Critic network. It is the loss function of the Critic network; the perturber receives the gradients passed from the Critic network. The perturbation is calculated using the FGSM method: ； in, It is the disturbance coefficient. It is the sign function, which takes the sign of the gradient; the perturbator adds this perturbation to the current state to obtain the adversarial sample: ； The Actor network receives the disturbance transmitted by the perturbator. It outputs actions based on the weight parameters of the Actor network. .