Intelligent vehicle platooning control method based on deep reinforcement learning under communication topology switching

Abstract

A deep reinforcement learning-based intelligent vehicle platooning control method under communication topology switching involves intelligent vehicle safety and autonomous driving. 1) Real-time acquisition of driving motion information from the ego vehicle, other following vehicles, and the pilot vehicle. 2) Analysis of the required vehicle control algorithms and algorithm optimization. Using graph theory to describe the information exchange between intelligent electric vehicles, an objective function for achieving platooning control is established. 3) Considering the system's communication failure scenario, a data fusion mechanism is established. In the event of a complete communication failure, an AKF is used to reduce perception errors and achieve communication topology switching for the multi-vehicle system. 4) Addressing the numerous drawbacks of traditional platooning system control, a deep reinforcement learning process is designed to iterate the platooning control strategy and obtain the optimal control strategy. This method achieves data-driven and communication topology switching for the platooning system, improving the controller's applicability and the overall vehicle performance, and addressing traffic congestion and safety concerns.

Description

Technical Field

[0001] The present invention belongs to the field of automobile intelligent safety and autonomous driving, and in particular relates to a method for controlling an intelligent vehicle formation based on deep reinforcement learning under communication topology switching. Background Art

[0002] With the rapid development of technology, the use of intelligent vehicles is also rapidly developing. As the number of vehicles continues to grow, traffic congestion is becoming increasingly serious, accompanied by severe energy consumption issues. Furthermore, the increasing number of vehicles increases the risk of traffic accidents, posing a significant threat to life and property. Intelligent vehicle longitudinal platooning control can reduce the distance between vehicles while ensuring safe driving, improve traffic efficiency, and reduce energy consumption, effectively alleviating these problems.

[0003] Intelligent vehicle longitudinal platoon control refers to the automatic adjustment of the longitudinal motion state of vehicles in a platoon according to the driving information of other vehicles, thereby ensuring that the platoon has the desired formation and consistent driving speed. Reference 1 (Li M, Cao Z, Li Z. Areinforcement learning-based vehicle platoon control strategy for reducing energy consumption in traffic oscillations [J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32 (12): 5309-5322.) proposes a data-driven form based on reinforcement learning model to realize vehicle platoon control. Reference 2 (Li K, Bian Y, Li SE, et al. Distributed model predictive control of multi-vehicle systems with switching communication topologies [J]. Transportation Research Part C: Emerging Technologies, 2020, 118: 102717.) proposes a distributed model-based predictive control method to solve the communication topology switching of multi-vehicle systems.

[0004] With the rapid and significant development of deep reinforcement learning (DRL), it has been applied as a data-driven control method in vehicle platooning systems, achieving excellent control performance. Compared to traditional control methods, data-driven control can cope with multiple traffic scenarios, does not require complex dynamic models, and has strong real-time performance. Data-driven control has achieved excellent control results in complex real-world traffic environments and scenarios with high real-time requirements. Vehicle-to-vehicle (V2V) communication can improve the safety and efficiency of vehicle platooning. However, communication failures such as packet loss, signal blocking, and hardware damage to vehicle communication modules can occur within the platoon. This can lead to switching of communication topologies between vehicles, thereby affecting the performance of the vehicle platooning system and reducing the robustness and applicability of the controller. Summary of the Invention

[0005] The purpose of the present invention is to address the many drawbacks of traditional vehicle formation controllers and communication topology switching problems in the prior art, and to provide a method for intelligent vehicle formation control based on deep reinforcement learning under communication topology switching. While taking into account the safety, comfort, and energy efficiency of the vehicles, the method ensures that all vehicles in the formation system maintain the desired spacing and speed, and realizes tracking control of the lead vehicle by the following vehicle. Based on a data fusion strategy, the present invention designs a control model based on deep reinforcement learning under communication topology switching for the vehicle longitudinal formation system, thereby training an efficient vehicle controller, which effectively solves the many drawbacks of poor controller applicability and poor real-time performance in traditional formation control, and realizes communication topology switching of the formation system in the event of a vehicle communication failure, thereby enhancing the applicable scenarios of the controller and achieving control performance indicators of formation control.

[0006] The present invention uses on-board sensors and a V2X wireless communication system to collect driving status information of both the vehicle itself and other vehicles. In the event of a communication failure, an inter-vehicle influence coefficient is designed. Weighted data fusion technology is used to compress and process the raw data, reducing and unifying the input layer dimension of the neural network. An adaptive Kalman filter (AKF) is also used to reduce the perception error of on-board sensors. A communication topology switching mechanism is established during the topology switching process. An improved deep deterministic policy gradient (DDPG) algorithm is used to solve the longitudinal control problem of the vehicle platoon. A dual-network structure is introduced based on the original DDPG algorithm to address the overestimation of the Q value of the neural network during model training. A prioritized experience replay mechanism is used to extract experience from the experience pool to address the sparse reward situation during the initial training phase. Considering safety, comfort, and following performance during driving, a reinforcement learning model and its multi-objective reward function are designed. A fully distributed controller based on deep reinforcement learning is trained under communication topology switching to achieve safe, comfortable, and energy-efficient driving for vehicle platoons.

[0007] The present invention comprises the following steps:

[0008] Step 1: The platoon consists of N+1 vehicles, numbered 0,…,N. Vehicle 0 is the lead vehicle, and vehicles 1,…,N are the following vehicles. Vehicle-to-vehicle sensors and the V2X wireless communication network collect real-time motion status information from the vehicle, the following vehicles, and the lead vehicle.

[0009] The first step is to periodically sense the vehicle's status information through on-board sensors and GPS, mainly including distance information, speed information, and acceleration information.

[0010] In the second step, the ego vehicle exchanges information with other vehicles in the formation through the V2X wireless communication network, and receives corresponding driving motion status information in real time, mainly including the driving distance and speed of the following vehicle and the leading vehicle, and receives the acceleration information of the leading vehicle at the same time.

[0011] Step 2 analyzes the algorithm type required for vehicle control and optimizes the shortcomings of the algorithm. Based on graph theory, it describes the information interaction between smart electric vehicles and establishes the objective function for smart electric vehicles to achieve platoon control.

[0012] The first step is to establish a data-driven control strategy to address the difficulties in modeling in complex actual traffic scenarios. This strategy takes into account the control types of the vehicle's throttle and brakes and selects a deep deterministic policy gradient algorithm.

[0013] In the second step, based on the original algorithm, a dual network structure is added to solve the problem of overestimation of the Q value of the valuation network. Considering the sparse rewards, a priority experience replay mechanism is adopted.

[0014] The third step is to define the communication topology of the vehicle formation based on graph theory, and give the corresponding definitions of the adjacency matrix and traction matrix; establish the objective function of vehicle formation control to keep the formation in a constant desired formation.

[0015] Step 3 considers the scenario of communication failure of the system and establishes a data fusion mechanism. In the case of complete communication failure, AKF is used to reduce perception errors and realize communication topology switching of the multi-vehicle system.

[0016] In the first step, the influence coefficient is given. Based on the communication support structure described by graph theory, the speed difference and distance difference between the ego vehicle and the communication vehicle are set as state variables, and the weighted average is used to establish a fusion mechanism.

[0017] In the second step, a dynamic communication topology model of the system is given, the influence of sensor perception error is considered, and the AKF method is used to reduce the perception error, thereby improving the accuracy of the vehicle control model.

[0018] The third step is to establish a smooth switching topology strategy to prevent the uneven control and sudden acceleration changes during the switching of communication topology, so as to achieve smooth driving of the vehicle formation.

[0019] Step 4: To address the many drawbacks of traditional formation system control, a deep reinforcement learning process is designed to iterate the formation control strategy and ultimately obtain the optimal control strategy.

[0020] In the first step, a reinforcement learning model is designed to update the vehicle state using a kinematic point-mass model.

[0021] The second step is to design a multi-objective reward function considering system performance goals such as safety and comfort, so as to achieve safe, comfortable and energy-saving driving of vehicle formations and improve the overall performance of the multi-vehicle system.

[0022] The third step is to establish a distributed formation control model, adjust the parameters of the neural network within the algorithm, train an effective controller, and calculate the expected output of the vehicle in real time.

[0023] The effects and benefits of the present invention are: designing a vehicle formation control method based on deep reinforcement learning under communication topology switching, compressing data dimensions by adopting data fusion, and designing a data-driven model using a deep reinforcement learning algorithm, adopting AKF in the model to reduce vehicle perception errors and introducing a method of smooth switching topology, thereby realizing data driving and communication topology switching of the formation system, greatly improving the applicability of the controller and the comprehensive driving performance of the vehicle, thereby significantly improving problems such as traffic congestion and safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the intelligent vehicle formation control framework of the present invention.

[0025] Figure 2 Schematic diagram of the intelligent vehicle platoon control system based on the improved DDPG under communication topology switching.

[0026] Figure 3 This is a flowchart of the present invention for training the network parameters of the improved DDPG algorithm. DETAILED DESCRIPTION

[0027] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0028] like Figure 1 As shown, the formation control system of the present invention adopts a deep reinforcement learning model to design a data-driven vehicle controller, which effectively solves the drawbacks of traditional methods; based on the communication topology structure, the driving information of other vehicles in the formation is obtained through on-board sensors and V2X wireless communication networks, and the dimension of the vehicle state space is compressed by weighted data fusion, so that the controller model can be applied to different topologies at the same time, realizing the communication topology switching of formation control, while reducing the dimension of the neural network input layer and improving the convergence speed of algorithm training. Relying on the above control method, a vehicle formation control method based on deep reinforcement learning under communication topology switching is realized. The specific steps include:

[0029] Step 1: A platoon consists of one lead vehicle and N follower vehicles, numbered 0, 1, ..., N. Each follower vehicle in the platoon uses onboard sensors and the V2X wireless communication network to collect real-time motion information from both the lead vehicle and the corresponding communication vehicle, based on its communication topology.

[0030] Step 1.1 Each following network electric vehicle continuously obtains its own driving information through on-board sensors and GPS, mainly including position information, speed information and acceleration information.

[0031] In step 1.2, based on different communication topologies, the ego vehicle communicates with other vehicles in the formation through V2X wireless communication network technology to collect corresponding vehicle driving status information in real time, including the driving position and speed information of the following vehicle or the leading vehicle.

[0032] Step 2: Use the improved DDPG algorithm as the control algorithm for the longitudinal cooperative control of the formation. Based on graph theory, describe the information interaction between smart electric vehicles and give the objective function of the smart electric vehicle formation control.

[0033] In step 2.1, considering that the vehicle's throttle and brake require continuous values ​​as control inputs, a policy-based reinforcement learning algorithm is used. Using the improved DDPG algorithm to solve the formation longitudinal control problem can effectively solve the formation control problem.

[0034] DRL can be modeled as a Markov decision process, which consists of two interacting objects: the DRL agent (control model) and the environment. The DRL framework has four basic elements: state s, action a, and strategy π. * and reward r, as shown in the following steps.

[0035] Step 2.2 Figure 2 As shown in the figure, based on the original DDPG algorithm, two critic networks are introduced to update the parameters, and the minimum target Q' value is selected as the update value of the critic target network. The target value formula generated by the two critic networks is as follows:

[0036]

[0037] Among them, r t+1 is the reward, γ is the discount factor, and is the exploration noise, Q'1 and Q'2 are the two target Q values.

[0038] In this algorithm, the TD-error value and critic network update value in reinforcement learning are as follows:

[0039]

[0040] y=min i＝1，2 y i

[0041] Where Q1 and Q2 are the current Q values ​​generated by the critic network.

[0042] To improve learning efficiency when rewards are sparse in the early stages, we introduce a prioritized experience replay (PER) mechanism, which relies on TD-error and real-time rewards to define the sampling probability of samples as follows:

[0043]

[0044] in, is a priority evaluation index, α represents the degree of priority, and ε is a constant.

[0045] Step 2.3 Based on graph theory, the vehicle communication topology is modeled as a directed time-varying graph corresponding to time t. A collection of nodes The set of edges consists of N following vehicles Describe the connectivity between vehicles. is a time-varying adjacency matrix. When vehicle i can receive information from vehicle j, c ij (t)=1, i≠j; otherwise c ij =0.

[0046] Assumptions in is the number of communication topology candidates. Definition Indicates communication with other following vehicles, including the number of the following vehicle, that is, Defining a traction collection If vehicle i can receive the pilot vehicle information, that is, q i (t) = 1; define a communication vehicle set

[0047] Given the control objectives of the vehicle formation, it is as follows:

[0048]

[0049] Among them, d i,i-1 is the desired constant time interval, p i is the position of the i-th vehicle.

[0050] Step 3: Apply a priori rules and incorporate the concept of balance into the vehicle-following model. Considering the system in unreliable communication scenarios, design the impact coefficient, establish a data fusion mechanism, implement communication topology switching of the multi-vehicle system, and design a topology switching mechanism.

[0051] Step 3.1 gives the concept of equilibrium when the vehicle formation reaches consistency and integrates the following strategy into the DRL algorithm framework, which is defined as follows:

[0052]

[0053] in, is the target equilibrium distance of vehicle i at time t, is the vehicle speed, is a constant time interval, l i is the static spacing,

[0054] The deviation between the actual vehicle distance and the target balance distance can be obtained from formula (5): And the relative speed is as follows:

[0055]

[0056] Step 3.2: Considering collision avoidance, design the influence coefficient based on the distance between vehicles. The preceding vehicle has the greatest impact on the safety of the ego vehicle. Use an exponential function to design the influence coefficient. The farther the other vehicles in the formation are from the ego vehicle, the smaller their impact, as shown below:

[0057]

[0058] Here, i is the vehicle number.

[0059] According to the influence coefficient and the set Establish a weighted data fusion mechanism:

[0060]

[0061] in, Represent the weighted position deviation and weighted speed deviation of vehicle i respectively, and obtain the state space of the reinforcement learning model

[0062] Step 3.3 In the scenario of unreliable wireless communication channels, there is the possibility of switching topology. The dynamic communication topology model of the system is established, which includes five communication topologies: predecessor following (PF), leader-predecessor following (LPF), two-predecessor following (TPF), two-predecessor-leader following (TPLF), and communication failure. After data processing, each is compressed into two data state spaces, namely the weighted deviations of spacing and speed, thereby unifying the input layers of neural networks under different topologies, accelerating algorithm convergence and better achieving the control objectives of the multi-vehicle system.

[0063] For vehicle i, based on the dynamic communication topology model, the driving information of the preceding vehicle i-1, the status information of vehicle i-2, and the pilot vehicle can be obtained through on-board sensors and V2X wireless communication technology. The status information of the control model is as follows:

[0064]

[0065]

[0066] However, when a vehicle in the formation loses its communication capability, that is, the vehicles cannot communicate with each other through V2X wireless communication technology, the vehicle can perceive the driving state of the preceding vehicle through sensors, and there will be perception errors accordingly. In order to improve the accuracy of sensor information, AKF is used to reduce the error and define a state vector It contains two perception state parameters, and the state equation and measurement equation are given as follows:

[0067] X t =AX t-1 +q t-1 ,q t-1 ~N(0,Q t-1 ) (11)

[0068] Z t =HX t +r t ,r t ~N(0,R t ) (12)

[0069] Among them, P t and V t are the position and velocity of the vehicle, is the state transition matrix, Represents the mapping relationship between the measured value and the true value, q t-1 and r t Denote random process noise and measurement noise respectively, assuming they have covariance matrix Q t and R t independent Gaussian noise components, and

[0070] The Kalman filter consists of two steps. The first step is the prior estimation. Since the vehicle tries to continuously perform the same driving action as the preceding vehicle as much as possible, its prior estimate is And its prior covariance matrix is Where P t-1 is the posterior covariance matrix,

[0071] The second step is the posterior estimation, which compares the actual measurement value with the predicted measurement value based on the prior estimation, and obtains the posterior state estimate after correction, as follows:

[0072]

[0073] Among them, K t is the Kalman gain, P t is the posterior covariance update equation, X t Indicates how to correct the estimated state based on the measured state to obtain a more accurate state.

[0074] To ensure accurate estimation, Q t and R t Make reasonable settings, R t is an inherent property of the vehicle, which is a constant value, and the fixed Q tThis will cause filtering deviation because it is automatically updated as follows:

[0075]

[0076] Where, ε t It's HX t The error between the measured value and the original value is α, which is the forgetting factor and is set to α = 0.3.

[0077] The formula of the Kalman filter-based predictor can be derived from equations (12), (13) and (14), as shown below:

[0078] X t =AX t-1 +(AP t-1 A T +Q t-1 )H T [H(AP t-1 A T +Q t-1 )H T +R t ] -1 (Z t -HAX t-1 ) (15)

[0079] Step 3.4: In order to reduce vehicle jolting during topology switching, a smooth transition method is added during the switching process. From Equation (7), it can be seen that each communication topology has a corresponding default influence coefficient. Assuming that the buffer time of the switching process is T, the linear change of the influence coefficient during this process is as follows:

[0080]

[0081] Where, t s is the start time of the switch, It is the default parameter after the switching is completed. If the communication between vehicles is lost, the value will be 0. are the default parameters at the start of the switch, where Assume T = 0.4.

[0082] When vehicle i and vehicle j lose communication, because the time T is very small, the driving information of the two vehicles in the previous T time can be used as buffer information during the switching process. When communication between vehicles is restored, the driving status of vehicle j can be obtained. When communication is lost, the buffer information of the motion estimation window size T is calculated as follows:

[0083]

[0084] Where, and are respectively the topology switching as (9) and (10)

[0085] Step 4: Design a deep reinforcement learning process to iterate the formation control strategy, considering vehicle safety, comfort, and energy efficiency, and design a multi-objective reward function.

[0086] Step 4.1 In the reinforcement learning model, the system environment needs to be continuously updated. After the vehicle receives the control command, the kinematic point-mass model is applied to update the vehicle state, as shown below:

[0087]

[0088]

[0089]

[0090] in, is the acceleration of the vehicle, given by the action space of the reinforcement learning model, let u = [a min ,a max ].

[0091] Step 4.2 first requires ensuring that the multi-vehicle system follows the nature of the platoon. The collaborative control efficiency cost and following vehicle reward are given as follows:

[0092]

[0093]

[0094] in, When controlling the cost, smaller control efficiency cost means higher following efficiency and local stability. α 1,i ,α 2,i >0 is a positive definite diagonal coefficient matrix, is the following car reward function.

[0095] Passenger comfort is an important indicator of vehicle driving performance. Jerk is defined as the rate of change of acceleration, and the following reward function is given:

[0096]

[0097] Safety is another important indicator of vehicle driving performance. The time integrated time-to-collision (TIT) is used to assess collision risk. The larger the TIT value, the higher the collision risk, as shown below:

[0098]

[0099]

[0100] Where, Indicates time to collision, TTC * represents the TTC threshold, T' is the calculation period, i is the vehicle number, and N is the total number of vehicles. The following reward function is given:

[0101]

[0102] Where τ is the current time headway of the vehicle, is a given value, let

[0103] Energy saving can reduce the fuel consumption of vehicles. The VT-Micro model and its reward function in the model are given as follows:

[0104]

[0105] Where, is the instantaneous fuel consumption rate (ml / s); K ij Polynomial coefficients.

[0106] From equations (20), (21), (23) and (24), we can see that the total reward function is:

[0107]

[0108] Where ξ1, ξ2, ξ3 and ξ4 are all fixed constants, and ξ1 = ξ2 = ξ3 = ξ4 = 1 is set.

[0109] Combined, we can know r i t For the reward function of the required multiple objectives, use the DRL strategy π * Formulate an infinite-horizon optimal control problem to maximize the discounted cumulative reward as follows:

[0110]

[0111] in, represents the reward function, k is the number of iterations, and γ is the discount factor.

[0112] Step 4.4 avoids dimensionality explosion in complex traffic scenarios, builds a distributed formation control model, adjusts the number of neural network nodes, weights, and hyperparameters in the reinforcement learning model, trains a DRL controller that meets the requirements, and obtains the expected output in real time.

[0113] Figure 3 The following is a flow chart of the training process of the improved DDPG algorithm.

[0114] The above content is a further detailed description of the present invention in conjunction with the preferred technical solution, and the specific implementation of the invention cannot be limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, simple deductions and substitutions can be made without departing from the concept of the present invention, which should be considered as the scope of protection of the present invention.

Claims

1. Intelligent vehicle platoon control method based on deep reinforcement learning under communication topology switching, characterized by The following steps are involved: 1) A platoon consists of N+1 vehicles, numbered 0,…,N, where vehicle 0 is the lead vehicle and vehicles 1,…,N are followers. Vehicle-to-vehicle sensors and the V2X wireless communication network collect real-time motion status information from the vehicle itself, other following vehicles, and the lead vehicle. 2) Analyze the types of algorithms required for vehicle control and optimize the drawbacks of the algorithms. Based on graph theory, describe the information interaction between smart electric vehicles and establish the objective function for smart electric vehicle platoon control. The specific steps include: (1) In view of the difficulty in modeling in complex actual traffic scenarios, a data-driven control strategy is established, taking into account the control types of the vehicle's throttle and brake, and selecting a deep deterministic policy gradient algorithm; (2) Based on the original algorithm, a dual network structure is added to solve the problem of overestimation of the Q value of the valuation network. Considering the sparse rewards, a priority experience replay mechanism is adopted; (3) Define the communication topology of the vehicle formation based on graph theory, and give the corresponding definitions of the adjacency matrix and traction matrix; establish the objective function of vehicle formation control to keep the formation in a constant desired formation; 3) Considering the scenario of system communication failure, a data fusion mechanism is established. In the event of a complete communication failure, AKF is used to reduce perception errors and implement communication topology switching for the multi-vehicle system. The specific steps include: (1) Given the influence coefficient, based on the communication support structure described by graph theory, the speed difference and distance difference between the ego vehicle and the communication vehicle are set as state variables, and the fusion mechanism is established using weighted average; (2) A dynamic communication topology model of the system is given. Considering the influence of sensor perception error, the AKF method is used to reduce the perception error, thereby improving the accuracy of the vehicle control model; (3) In order to prevent the uneven control and sudden acceleration changes in the process of switching communication topology, a smooth switching topology strategy is established to achieve smooth driving of the vehicle formation; 4) To address the many drawbacks of traditional formation control systems, a deep reinforcement learning process is designed to iterate the formation control strategy and ultimately obtain the optimal control strategy; The specific steps of designing a deep reinforcement learning process to iterate the formation control strategy and ultimately obtain the optimal control strategy include: (1) Design a reinforcement learning model and apply the kinematic point-mass model to update the vehicle state; (2) Considering the safety and comfort system performance goals, a multi-objective reward function is designed to achieve safe, comfortable, and energy-efficient driving of vehicle formations and improve the overall performance of the multi-vehicle system; (3) Establish a distributed formation control model, adjust the parameters of the neural network within the algorithm, train the controller, and calculate the expected output of the vehicle in real time.

2. The intelligent vehicle platoon control method based on deep reinforcement learning under communication topology switching as described in claim 1 is characterized in that In step 1), the real-time collection of the driving motion status information of the vehicle and other following vehicles and the pilot vehicle includes: (1) Periodically sense the vehicle's status information, including distance, speed, and acceleration, through on-board sensors and GPS; (2) The ego vehicle exchanges information with other vehicles in the formation through the V2X wireless communication network, and receives corresponding driving motion status information in real time, including the driving distance and driving speed of the following vehicle and the leading vehicle, and receives the acceleration information of the leading vehicle at the same time.

Images