Vehicle cooperative control method and system in ramp confluence area scene in mixed traffic

Abstract

The invention discloses a vehicle cooperative control method and system in a ramp confluence area scene in mixed traffic, and relates to the technical field of intelligent traffic. Comprising the following steps: constructing traffic scenes of a single-lane main road and a single-lane ramp, and designing a state space, an action space and a reward function corresponding to each agent; designing a double-delay depth deterministic strategy gradient ITD3 model, and controlling the CAV to perform two-dimensional continuous transverse driving and longitudinal driving; fusing a two-dimensional collision avoidance strategy for each agent, and constructing a KBS-ITD3 model by fusing a double-delay depth deterministic strategy gradient ITD3 model; a network model corresponding to the KBS-ITD3 model is established for training, and the trained KBS-ITD3 model is used for controlling vehicles in a ramp confluence area scene. The safety in the confluence process is greatly improved while the efficiency is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and more specifically to a vehicle cooperative control method and system for a ramp merging area scene in mixed traffic. Background Art

[0002] As intelligent vehicles with precise perception, communication and interaction capabilities, connected vehicles (CAVs) are expected to significantly improve road safety, ease traffic congestion, and fundamentally change traditional travel modes and traffic management models. With the development and popularization of CAV technology, mixed traffic environments where CAVs and human-driven vehicles (HDVs) coexist will become more and more common. Complex traffic environments and challenging driving scenarios have brought higher requirements to CAVs in terms of safety, intelligence and efficiency. Among many challenging driving scenarios, highway entrance ramp merging is one of the most difficult tasks for autonomous vehicles. Vehicles from the upstream of the main line meet vehicles merging into the entrance ramp here. Different driving behaviors and high traffic volume lead to significant fluctuations in traffic flow in the merging area, which is very likely to cause congestion and traffic accidents.

[0003] Most previous studies have solved the CAV merging problem based on rule-based and optimization methods. Rule-based methods usually solve the problem through pre-coded rules, so they can provide fast and easy-to-implement solutions. However, when dealing with complex and dynamic traffic environments, such methods are difficult to adapt to different traffic flows and emergencies due to the one-sidedness and lack of flexibility of the rules, and collision accidents are very likely to occur. Optimization-based methods usually involve minimizing the cost function on a constrained sequence of state variables. However, as the number of vehicles increases and the road conditions become more complex, the computational cost becomes particularly prominent, resulting in a decrease in the feasibility of real-time applications.

[0004] The multi-intelligence deep reinforcement learning (MADRL) method is regarded as a new direction in solving complex tasks of autonomous driving. Various MADRL-based methods have been applied to solve the CAV ramp merging problem. However, in the current research on CAV control in ramp scenarios, most of the lane changing process is ignored or simplified, resulting in simplified vehicle interaction between different lanes. Under normal circumstances, only considering separate longitudinal control or using a fixed lateral speed or transient lane changing process is difficult to reflect the real traffic dynamics. In addition, the effectiveness of the MADRL algorithm depends largely on extensive training data, and there are major problems with the learning efficiency of the algorithm. At the same time, as a black box method, MADRL is often criticized for its lack of interpretability, which may lead to safety issues in extreme situations. Therefore, it is impossible to efficiently and stably solve the problem of vehicle collaborative control in complex entrance ramp scenarios.

[0005] Therefore, it is an urgent problem for those skilled in the art to propose a vehicle cooperative control method and system for a ramp merging area scenario in mixed traffic to solve the difficulties existing in the prior art. Summary of the invention

[0006] In view of this, the present invention provides a vehicle collaborative control method and system for a ramp merging area scenario in mixed traffic, so as to solve the technical problem that the existing methods cannot efficiently, stably and accurately realize the collaborative control of intelligent connected vehicles (CAVs) in ramp merging scenarios under mixed traffic flows.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A vehicle cooperative control method for a ramp merging area scenario in mixed traffic comprises the following steps:

[0009] Construct traffic scenarios of single-lane main roads and single-lane ramps, and design the state space, action space, and reward function corresponding to each agent;

[0010] Design a dual-delay deep deterministic policy gradient ITD3 model to control CAV to perform two-dimensional continuous lateral and longitudinal driving;

[0011] A two-dimensional collision avoidance strategy is integrated for each agent, and a KBS-ITD3 model is constructed by incorporating a double-delayed deep deterministic policy gradient ITD3 model.

[0012] By building a corresponding network model for the KBS-ITD3 model and training it, the trained KBS-ITD3 model is used to control vehicles in the ramp merging area scenario.

[0013] Optionally, the specific content of the state space corresponding to each agent is designed as follows:

[0014] The state space of each agent is defined as a matrix of size N×W, where N is the number of observed vehicles and W is the number of features used to represent the state of each vehicle. Each vehicle feature consists of the following seven elements:

[0015] W = {ispresence,x,y,v lon ,v lat ,a lon ,a lat}

[0016] Among them, ispresence is whether the nearest vehicle is within the perception range of the current vehicle; x is the real-time x coordinate of the nearest vehicle; y is the real-time y coordinate of the nearest vehicle; v lon is the real-time longitudinal velocity of the nearest vehicle; v lat is the real-time lateral speed of the nearest vehicle; alon is the most recent real-time longitudinal acceleration of the vehicle; a lat It is the real-time lateral acceleration of the nearest vehicle.

[0017] Optionally, the specific content of the action space corresponding to each agent is designed as follows:

[0018] In order to couple the longitudinal and lateral motions of the vehicle, the action space is defined as a two-dimensional continuous space consisting of longitudinal acceleration and lateral acceleration. The CAV adjusts its speed by continuously performing forward and steering operations.

[0019] Optionally, the specific content of the reward function corresponding to each agent is designed as follows:

[0020] The collision reward function r of the vehicle c for:

[0021]

[0022] If a collision occurs, the corresponding CAV collision assessment index r c will be set to -200, otherwise r c The value of is 0;

[0023] The vehicle's speed reward function r s for:

[0024]

[0025] Among them, v lon (t) is the longitudinal velocity of the vehicle at time t; v lat (t) is the lateral speed of the vehicle at time t; vmin is the threshold of the minimum speed of the vehicle; v max is the threshold value of the maximum speed of the vehicle;

[0026] The vehicle's merging distance reward function r d The definition is as follows:

[0027]

[0028] Where d is the distance the vehicle travels in the acceleration lane, and L is the length of the acceleration lane. As the distance the vehicle travels in the acceleration lane increases, the penalty value will gradually increase.

[0029] The risk assessment reward function r of the vehicle t for:

[0030]

[0031] in, The setting value is 5.0s. 2DThreshold, when it is lower than 5.0s, CAV will be penalized; at the same time, the minimum penalty value is set to -1; T 2D In order to design a two-dimensional collision time index for risk assessment during the merging process, the calculation formula is as follows:

[0032]

[0033] Among them, x1 is the real-time x coordinate of the current vehicle; x2 is the real-time x coordinate of the vehicle in front; y1 is the real-time y coordinate of the current vehicle; y2 is the real-time y coordinate of the vehicle in front; d 12 is the distance between the center of the current vehicle and the vehicle in front; l is the length of the current vehicle, β1 is the steering angle of the current vehicle; β2 is the steering angle of the vehicle in front; α is the angle between the current vehicle and the vehicle in front; ν1 is the speed of the current vehicle; ν2 is the speed of the vehicle in front;

[0034] Taking all the above elements into account, the total reward function is given by the weighted sum of all rewards:

[0035]

[0036] Among them, w c 、w t 、w s and w d are the weights of the different components of the reward function. After parameter adjustment for the weighting coefficients of the four different driving characteristics, their values ​​are as follows: c =1, w d =1, w s =2 and w t =4.

[0037] Optionally, the specific contents of designing the dual-delay deep deterministic policy gradient ITD3 model are:

[0038] Using a distributed learning method, the dual-delay deep deterministic policy gradient ITD3 model based on the Actor-Critic framework is expanded to the multi-agent level, and parameter sharing is added to control vehicle driving. The critic network update parameter will select the smallest Q value calculation error in the two critic target networks, that is:

[0039]

[0040] Among them, γ is the discount coefficient, ε is the added random noise, and s t is the current state, s t ' is the next state, a t is the current action, a t ′ is the target action, r t is the reward value obtained, is the Critic target network, To calculate the gradient of the Critic target network parameters, it follows the truncated normal distribution clip(N(0,σ),-c,c),c>0;

[0041] The parameters of the Actor network and the three target networks are updated as follows:

[0042]

[0043] Where τ is the soft update coefficient, To update the parameters of the Actor network, To calculate the gradient of the Actor network parameters, For the Critic network, is the Actor network, s t is the extracted state, θ i ′ is the Critic target network parameter, is the Actor target network parameter, Actor network parameters.

[0044] Optionally, the specific content of the two-dimensional collision avoidance strategy for each agent is:

[0045] The collision avoidance strategy is to avoid a collision by driving the current vehicle at a safe stopping speed even if the vehicle ahead brakes suddenly. The Krauss model is used to calculate the safe speed using the following formula:

[0046]

[0047] Among them, v n (t) is the longitudinal velocity of the current vehicle n at time t; v n-1 (t) is the longitudinal speed of the front vehicle n-1 at time t; g(t) is the distance between the current vehicle and the front vehicle at time t; b is the maximum deceleration of the current vehicle, which is set to -4.5 m / s 2 ; τ is the driver's reaction time set to 0.5s;

[0048] The collision avoidance strategy uses the following formula:

[0049]

[0050] Among them, v lon (t+1) is the longitudinal velocity of the next time step calculated according to the longitudinal acceleration output of the double-delayed deep deterministic policy gradient ITD3 model at time t; v safe (t+1) is the safe speed calculated at time t according to the Krauss model; It is T2D The maximum threshold is set to 2.5s.

[0051] A vehicle cooperative control system for a ramp merging area scenario in mixed traffic, a vehicle cooperative control method for a ramp merging area scenario in mixed traffic using any of the above items, comprising a simulation module, a state acquisition module and a control module connected in sequence; wherein,

[0052] The simulation module builds a running motion simulation model based on the vehicle's dynamics model;

[0053] The state acquisition module acquires the state space and action space of the vehicle platoon control based on the motion simulation model of the vehicle platoon operation and the vehicle operation status using the constant time spacing strategy;

[0054] The control module establishes a corresponding network based on the state space and action space of the CAV, optimizes and trains the designed network according to the reward function mechanism, obtains the control strategy network, and uses the trained control strategy network to control the vehicle in the ramp merging scenario.

[0055] Optionally, a corresponding network is established to optimize the design network according to the reward function mechanism. During the training process, different numbers of CAVs and HDVs will be randomly generated from the ramp and main road positions in each round, and random position noise will be added to each initial generated position.

[0056] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a vehicle cooperative control method and system for a ramp merging area scenario in mixed traffic, and its beneficial effects are:

[0057] 1) Starting from the ramp merging area scenario where intelligent connected vehicles (CAVs) and human-driven vehicles (HDVs) coexist, interpretable structured features are set as state space inputs, reward functions are set from multiple perspectives such as safety, comfort, and risk assessment, and the two-dimensional continuous action space of the vehicle is considered as the output, simulating the high-precision and high-reliability driving process in reality;

[0058] 2) Distributed learning is used to expand the dual-delay deep deterministic policy gradient ITD3 model to the multi-agent level. Each intelligent connected vehicle CAV is regarded as a separate agent, and all other agents are regarded as part of the environment. In situations involving collaboration and interaction between multiple agents, the mutual interference and complexity between multiple agents are effectively reduced, and collision avoidance strategies are designed. This greatly improves the safety of the merging process while ensuring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 A flow chart of a vehicle cooperative control method for a ramp merging area scenario in mixed traffic provided by the present invention;

[0061] Figure 2 A structural diagram of a vehicle cooperative control system for a ramp merging area scenario in mixed traffic provided by the present invention;

[0062] Figure 3 A schematic diagram of a mixed traffic ramp merging area scenario provided by an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of two-dimensional collision time calculation provided by an embodiment of the present invention;

[0064] Figure 5 This is a diagram of the overall network architecture provided by the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] For environmental interaction, the present invention uses a merging area with an entrance ramp and a main lane as a research scenario, such as Figure 3 As shown in the figure, CAV and HDV coexist in this scenario. Ideally, ramp vehicles need to make autonomous lateral and longitudinal decisions to merge into the straight main lane efficiently and safely; vehicles on the straight main lane should coordinate to provide sufficient merging space for entrance ramp vehicles to avoid collision accidents and improve traffic efficiency.

[0067] See also Figure 1 As shown, the present invention discloses a vehicle cooperative control method for a ramp merging area scene in mixed traffic, comprising the following steps:

[0068] Construct traffic scenarios of single-lane main roads and single-lane ramps, and design the state space, action space, and reward function corresponding to each agent;

[0069] Design a dual-delay deep deterministic policy gradient ITD3 model to control CAV to perform two-dimensional continuous lateral and longitudinal driving;

[0070] A two-dimensional collision avoidance strategy is integrated for each agent, and a KBS-ITD3 model is constructed by incorporating a double-delayed deep deterministic policy gradient ITD3 model.

[0071] By building a corresponding network model for the KBS-ITD3 model and training it, the trained KBS-ITD3 model is used to control vehicles in the ramp merging area scenario.

[0072] Furthermore, the specific content of the state space corresponding to each agent is designed as follows:

[0073] The state space of each agent is defined as a matrix of size N×W, where N is the number of observed vehicles, the current vehicle and the four nearest vehicles are selected as the number of observed vehicles, and W is the number of features used to represent the state of each vehicle. Each vehicle feature consists of the following seven elements:

[0074] W = {ispresence,x,y,v lon ,v lat ,a lon ,a lat}

[0075] Among them, ispresence is whether the nearest vehicle is within the perception range of the current vehicle; x is the real-time x coordinate of the nearest vehicle; y is the real-time y coordinate of the nearest vehicle; v lon is the real-time longitudinal velocity of the nearest vehicle; v lat is the real-time lateral speed of the nearest vehicle; a lon is the most recent real-time longitudinal acceleration of the vehicle; a lat It is the real-time lateral acceleration of the nearest vehicle.

[0076] Specifically, for this environment, a corresponding state space is designed for each agent, using interpretable structured features as input. Based on the sensing equipment such as the laser radar on the vehicle body, the relevant state of the nearest vehicle within 100m from each agent can be provided.

[0077] Furthermore, the specific content of the action space corresponding to each agent is designed as follows:

[0078] In order to couple the longitudinal and lateral motions of the vehicle, the action space is defined as a two-dimensional continuous space consisting of longitudinal acceleration and lateral acceleration. The CAV adjusts its speed by continuously performing forward and steering operations.

[0079] Specifically, the longitudinal acceleration and lateral acceleration ranges of the vehicle are set to [-3, 3] m / s2 and [-1, 1] m / s 2 The lower-level controller of the autonomous vehicle combines the vehicle kinematic model to convert the two-dimensional actions output by reinforcement learning into appropriate lateral and longitudinal vehicle speed control.

[0080] Furthermore, the reward function of each agent is designed from multiple perspectives such as security and efficiency. The specific content of the reward function corresponding to each agent is designed as follows:

[0081] The collision reward function r of the vehicle c for:

[0082]

[0083] If a collision occurs, the corresponding CAV collision assessment index r c will be set to -200, otherwise r c The value of is 0;

[0084] Specifically, since safety is the most important criterion, when a collision occurs c The penalty value is large, so safety can be prioritized.

[0085] The vehicle's speed reward function r s for:

[0086]

[0087] Among them, v lon (t) is the longitudinal velocity of the vehicle at time t; v lat (t) is the lateral velocity of the vehicle at time t; v min is the threshold value of the minimum speed of the vehicle; v max is the threshold value of the maximum speed of the vehicle;

[0088] The vehicle's merging distance reward function r d The definition is as follows:

[0089]

[0090] Where d is the distance the vehicle travels in the acceleration lane, and L is the length of the acceleration lane. As the distance the vehicle travels in the acceleration lane increases, the penalty value will gradually increase.

[0091] The risk assessment reward function r of the vehicle t for:

[0092]

[0093] in, The setting value is 5.0s. 2DThreshold, when it is lower than 5.0s, CAV will be penalized; at the same time, the minimum penalty value is set to -1; T 2D For the design of a two-dimensional collision time metric for risk assessment during merging, see Figure 4 As shown, the calculation formula is as follows:

[0094]

[0095] Among them, x1 is the real-time x coordinate of the current vehicle; x2 is the real-time x coordinate of the vehicle in front; y1 is the real-time y coordinate of the current vehicle; y2 is the real-time y coordinate of the vehicle in front; d 12 is the distance between the center of the current vehicle and the vehicle in front; l is the length of the current vehicle, β1 is the steering angle of the current vehicle; β2 is the steering angle of the vehicle in front; α is the angle between the current vehicle and the vehicle in front; ν1 is the speed of the current vehicle; ν2 is the speed of the vehicle in front;

[0096] Specifically, to avoid T 2D If the value is too small, the penalty will be too large. Set the minimum penalty value to -1.

[0097] Taking all the above elements into account, the total reward function is given by the weighted sum of all rewards:

[0098]

[0099] Among them, w c 、w t 、w s and w d are the weights of the different components of the reward function. After parameter adjustment for the weighting coefficients of the four different driving characteristics, their values ​​are as follows: c =1, w d =1, w s =2 and w t =4.

[0100] For further information, see Figure 5 As shown in the figure, the specific contents of designing the dual-delay deep deterministic policy gradient ITD3 model are:

[0101] Specifically, the MADRL problem of n agents is decomposed into n single-agent problems, where each agent learns independently and only considers its own actions, while considering other agents as part of the environment;

[0102] Each agent requires six networks: Actor Network and parameters are θ i＝1 and θ i＝2 Critic Network and And the parameters are Actor target network The parameters are θ i ' ＝1 and θ i ' ＝2 Critic target network and Actor Network and Actor target network According to the current state s t and the next state s t ' Output current action a t and target action a t ′.

[0103] Using a distributed learning method, the dual-delay deep deterministic policy gradient ITD3 model based on the Actor-Critic framework is expanded to the multi-agent level, and parameter sharing is added to control vehicle driving. The critic network update parameter will select the smallest Q value calculation error in the two critic target networks, that is:

[0104]

[0105] Among them, γ is the discount coefficient, ε is the added random noise, and s t is the current state, s t ' is the next state, a t is the current action, a t ′ is the target action, r t is the reward value obtained, is the Critic target network, To calculate the gradient of the Critic target network parameters, it follows the truncated normal distribution clip(N(0,σ),-c,c),c>0;

[0106] The parameters of the Actor network and the three target networks are updated as follows:

[0107]

[0108] Where τ is the soft update coefficient, To update the parameters of the Actor network, To calculate the gradient of the Actor network parameters, For the Critic network, is the Actor network, s t is the extracted state, θ i ′ is the Critic target network parameter, is the Actor target network parameter, Actor network parameters.

[0109] Specifically, at the same time, compared with the update of the action value function parameters of the Critic part every step, the update frequency of the policy network parameters of the Actor part is lower, usually the update frequency is greater than 2, that is, it is updated once every 2 steps. In order to accelerate network convergence and improve the learning efficiency of the agent, parameter sharing technology is added. By multiple agents sharing the same set of parameters, the strategies of each agent can influence and adjust each other, effectively improving the stability of the strategy. At the same time, an adaptive double buffer mechanism based on success rate is provided to improve the utilization of effective experience. By dividing the experience pool into M success and M failure , to classify and store different types of experience data.

[0110] Specifically, during training, based on the current merging success rate, some experience data is selected from each playback buffer as samples according to different probabilities. Deep reinforcement learning has a time delay, and past experience often affects the current state of the CAV. Therefore, in addition to the experience pool M success and M failure In addition, a temporary experience pool M is set up for each CAV temp , used to store all driving experiences of the CAV in this round. Assuming that at time t, the CAV fails to merge in this round, and the driving experience obtained at this time is e, it can be determined that the driving experience e at time t-α has a negative impact on the CAV and is a failed driving experience. CAV failure experience is divided into two types: collision during driving and failure to reach the destination after exceeding the scheduled time. When the CAV fails due to collision, a is set to 25; when the CAV fails due to timeout, a is set to 50.

[0111] When CAV completes this merging task, if the result is a failure, the corresponding experience pool M temp The driving experience within the time [ta, t] is stored in the experience pool M failure , the rest of the experience is stored in the experience pool M success ; If the result is successful, all experience will be stored in experience pool M success .

[0112] During training, different amounts of sample data will be selected from the two buffers, as shown in the following formula:

[0113]

[0114] Among them, N failure and N success From the experience pool M failure and M successThe size of the sampled data in the experiment; β is the success rate in the last 200 rounds; M is the batch size; λ is the discount coefficient added before the number of failed experiences is selected. To prevent too many failed experiences from being extracted, λ is taken as 0.25 in the experiment; γ is the constant term added after the failed experiences. To prevent too few failed experiences from being extracted and to ensure the diversity of the sample data, γ is taken as 5.

[0115] Furthermore, a method for invalid action shielding is designed to avoid collecting meaningless and misleading experience during training. Invalid actions output by the Actor network are shielded and replaced by correct actions obtained by recalculation. Consider the following invalid operations:

[0116] The vehicle is trying to go beyond the lane. For example, the vehicle is already on the far right side of the road but is trying to accelerate to the right.

[0117] When the acceleration is output, if the calculated speed exceeds the maximum speed of the current vehicle or is lower than the minimum speed, the appropriate acceleration will be recalculated.

[0118] Furthermore, the specific content of the two-dimensional collision avoidance strategy for each agent is as follows:

[0119] The collision avoidance strategy is to avoid a collision by driving the current vehicle at a safe stopping speed even if the vehicle ahead brakes suddenly. The Krauss model is used to calculate the safe speed using the following formula:

[0120]

[0121] Among them, v n (t) is the longitudinal velocity of the current vehicle n at time t; v n-1 (t) is the longitudinal speed of the front vehicle n-1 at time t; g(t) is the distance between the current vehicle and the front vehicle at time t; b is the maximum deceleration of the current vehicle, which is set to -4.5 m / s 2 ; τ is the driver's reaction time set to 0.5s;

[0122] Specifically, the Krauss model emphasizes collision-free driving, so the calculated safe speed is very conservative. Overly cautious driving will aggravate the stop-and-go phenomenon. Therefore, the design of the collision avoidance strategy is only based on the real-time two-dimensional collision time T of the vehicle. 2D The current vehicle will identify the vehicle closest to its front bumper as the vehicle ahead to calculate Krauss's safe speed.

[0123] The collision avoidance strategy uses the following formula:

[0124]

[0125] Among them, v lon (t+1) is the longitudinal velocity of the next time step calculated according to the longitudinal acceleration output of the double-delayed deep deterministic policy gradient ITD3 model at time t; v safe (t+1) is the safe speed calculated at time t according to the Krauss model; It is T 2D The maximum threshold is set to 2.5s.

[0126] Specifically, through this collision avoidance strategy, unreasonable behaviors can be shielded, and faster model convergence can be achieved while ensuring vehicle safety.

[0127] The present invention also provides a vehicle cooperative control system for a ramp merging area scene in mixed traffic, which is used to Figure 1 The specific implementation of the method in the above example is shown in the schematic diagram of its structure. Figure 2 As shown, it includes a simulation module, a state acquisition module and a control module connected in sequence; wherein,

[0128] The simulation module builds a running motion simulation model based on the vehicle's dynamics model;

[0129] The state acquisition module acquires the state space and action space of the vehicle platoon control based on the motion simulation model of the vehicle platoon operation and the vehicle operation status using the constant time spacing strategy;

[0130] The control module establishes a corresponding network based on the state space and action space of the CAV, optimizes and trains the designed network according to the reward function mechanism, obtains the control strategy network, and uses the trained control strategy network to control the vehicle in the ramp merging scenario.

[0131] Specifically, after the training process, it can integrate the advantages of multi-agent deep reinforcement learning and collision avoidance strategy, and obtain a vehicle cooperative control network model for entrance ramp scenarios in mixed traffic with fast convergence speed, strong decision-making ability and high safety assurance, so as to perform vehicle control in ramp merging scenarios.

[0132] Furthermore, a corresponding network is established to optimize the designed network according to the reward function mechanism. During the training process, different numbers of CAVs and HDVs will be randomly generated from the ramp and main road positions in each round, and random position noise will be added to each initial generated position.

[0133] Specifically, the initial velocity will be randomly selected from 18m / s to 23m / s, and the maximum velocity of CAV is 30m / s. The simulation step size is 0.2s. 5% random noise is added to the lateral and longitudinal velocities and accelerations in the CAV observation state. The lateral and longitudinal models of HDVs are SL2015 and IDM, respectively, and the motion of CAVs is controlled by the KBS-ITD3 model. Table 1 lists the hyperparameters of the used models, see Table 1.

[0134] Table 1 Key hyperparameters for model training

[0135] parameter Setting Value The first hidden layer 256 The second hidden layer 128 Activation Function ReLU Actor network learning rate 0.0005 Critic network learning rate 0.0005 Discount factor γ 0.99 Soft update rate τ 0.005 Maximum round step length 500 Total rounds 10000 Sample size 256

[0136] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0137] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle cooperative control method for a ramp merging area in mixed traffic, characterized in that: The following steps are involved: Construct traffic scenarios of single-lane main roads and single-lane ramps, and design the state space, action space, and reward function corresponding to each agent; Design a dual-delay deep deterministic policy gradient ITD3 model to control CAV to perform two-dimensional continuous lateral and longitudinal driving; A two-dimensional collision avoidance strategy is integrated for each agent, and a KBS-ITD3 model is constructed by incorporating a double-delayed deep deterministic policy gradient ITD3 model. By building a corresponding network model for the KBS-ITD3 model and training it, the trained KBS-ITD3 model is used to control vehicles in the ramp merging area scenario.

2. The vehicle cooperative control method for a ramp merging area scene in mixed traffic according to claim 1 is characterized in that: The specific content of the state space corresponding to each agent is designed as follows: The state space of each agent is defined as a matrix of size N×W, where N is the number of observed vehicles and W is the number of features used to represent the state of each vehicle. Each vehicle feature consists of the following seven elements: W={ispresence,x,y,v lon ,v lat ,a lon ,a lat } Among them, ispresence is whether the nearest vehicle is within the perception range of the current vehicle; x is the real-time x coordinate of the nearest vehicle; y is the real-time y coordinate of the nearest vehicle; v lon is the real-time longitudinal velocity of the nearest vehicle; v lat is the real-time lateral speed of the nearest vehicle; a lon is the most recent real-time longitudinal acceleration of the vehicle; a lat It is the real-time lateral acceleration of the nearest vehicle.

3. The vehicle cooperative control method for a ramp merging area scene in mixed traffic according to claim 1 is characterized in that: The specific content of designing the action space corresponding to each agent is: In order to couple the longitudinal and lateral motions of the vehicle, the action space is defined as a two-dimensional continuous space consisting of longitudinal acceleration and lateral acceleration. The CAV adjusts its speed by continuously performing forward and steering operations.

4. The vehicle cooperative control method for a ramp merging area in mixed traffic according to claim 1, characterized in that: The specific content of designing the reward function corresponding to each agent is: The collision reward function r of the vehicle c for: If a collision occurs, the corresponding CAV collision assessment index r c will be set to -200, otherwise r c The value of is 0; The vehicle's speed reward function r s for: Among them, v lon (t) is the longitudinal velocity of the vehicle at time t; v lat (t) is the lateral velocity of the vehicle at time t; v min is the threshold value of the minimum speed of the vehicle; v max is the threshold value of the maximum speed of the vehicle; The vehicle's merging distance reward function r d The definition is as follows: Where d is the distance the vehicle travels in the acceleration lane, and L is the length of the acceleration lane. As the distance the vehicle travels in the acceleration lane increases, the penalty value will gradually increase. The risk assessment reward function r of the vehicle t for: in, The setting value is 5.0s. 2D Threshold, when it is lower than 5.0s, CAV will be penalized; at the same time, the minimum penalty value is set to -1; T 2D In order to design a two-dimensional collision time index for risk assessment during the merging process, the calculation formula is as follows: Among them, x1 is the real-time x coordinate of the current vehicle; x2 is the real-time x coordinate of the vehicle in front; y1 is the real-time y coordinate of the current vehicle; y2 is the real-time y coordinate of the vehicle in front; d 12 is the distance between the center of the current vehicle and the vehicle in front; l is the length of the current vehicle, β1 is the steering angle of the current vehicle; β2 is the steering angle of the vehicle in front; α is the angle between the current vehicle and the vehicle in front; ν1 is the speed of the current vehicle; ν2 is the speed of the vehicle in front; Taking all the above elements into account, the total reward function is given by the weighted sum of all rewards: Among them, w c 、w t 、w s and w d are the weights of the different components of the reward function. After parameter adjustment for the weighting coefficients of the four different driving characteristics, their values ​​are as follows: c =1, w d =1, w s =2 and w t =4.

5. The vehicle cooperative control method for a ramp merging area scene in mixed traffic according to claim 1 is characterized in that: The specific contents of designing the dual-delay deep deterministic policy gradient ITD3 model are: Using a distributed learning method, the dual-delay deep deterministic policy gradient ITD3 model based on the Actor-Critic framework is expanded to the multi-agent level, and parameter sharing is added to control vehicle driving. The critic network update parameter will select the smallest Q value calculation error in the two critic target networks, that is: Among them, γ is the discount coefficient, ε is the added random noise, and s t is the current state, s t ' is the next state, a t is the current action, a t ′ is the target action, r t is the reward value obtained, is the Critic target network, To calculate the gradient of the Critic target network parameters, it follows the truncated normal distribution clip(N(0,σ),-c,c),c>0; The parameters of the Actor network and the three target networks are updated as follows: Where τ is the soft update coefficient, To update the parameters of the Actor network, To calculate the gradient of the Actor network parameters, For the Critic network, is the Actor network, s t is the extracted state, θ i ′ is the Critic target network parameter, is the Actor target network parameter, Actor network parameters.

6. The vehicle cooperative control method for a ramp merging area in mixed traffic according to claim 1, characterized in that: The specific content of integrating the two-dimensional collision avoidance strategy for each agent is: The collision avoidance strategy is to avoid a collision by driving the current vehicle at a safe stopping speed even if the vehicle ahead brakes suddenly. The Krauss model is used to calculate the safe speed using the following formula: Among them, v n (t) is the longitudinal velocity of the current vehicle n at time t; v n-1 (t) is the longitudinal speed of the front vehicle n-1 at time t; g(t) is the distance between the current vehicle and the front vehicle at time t; b is the maximum deceleration of the current vehicle, which is set to -4.5 m / s 2 ; τ is the driver's reaction time set to 0.5s; The collision avoidance strategy uses the following formula: Among them, v lon (t+1) is the longitudinal velocity of the next time step calculated according to the longitudinal acceleration output of the double-delayed deep deterministic policy gradient ITD3 model at time t; v safe (t+1) is the safe speed calculated at time t according to the Krauss model; It is T 2D The maximum threshold is set to 2.5s.

7. A vehicle cooperative control system for a ramp merging area in mixed traffic, characterized in that A vehicle cooperative control method for a ramp merging area scenario in mixed traffic according to any one of claims 1 to 6, comprising a simulation module, a state acquisition module and a control module connected in sequence; wherein: The simulation module builds a running motion simulation model based on the vehicle's dynamics model; The state acquisition module acquires the state space and action space of the vehicle platoon control based on the motion simulation model of the vehicle platoon operation and the vehicle operation status using the constant time spacing strategy; The control module establishes a corresponding network based on the state space and action space of the CAV, optimizes and trains the designed network according to the reward function mechanism, obtains the control strategy network, and uses the trained control strategy network to control the vehicle in the ramp merging scenario.

8. The vehicle cooperative control system for a ramp merging area in mixed traffic according to claim 7, characterized in that: A corresponding network is established to optimize the designed network according to the reward function mechanism. During the training process, different numbers of CAVs and HDVs will be randomly generated from the ramp and main road positions in each round, and random position noise will be added to each initial generated position.

Images