Vehicle control method for mainline multilane highway ramp merging area based on deep reinforcement learning

Abstract

The application relates to the technical field of intelligent transportation, and discloses a main line multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning, which comprises the following steps: acquiring the motion state of a main line intelligent agent and a ramp intelligent agent of a main line multi-lane highway ramp merging area under a mixed traffic flow; based on a multi-agent advantage actor-critic method with stable policy gradient, the motion state of a main line indefinite number of agents is adjusted in real time; based on a single-agent flexible actor-critic method with continuous action space exploration, the motion state of the ramp intelligent agent is dynamically updated; a heterogeneous intelligent agent cooperative control framework is constructed with the main line intelligent agent and the ramp intelligent agent; the expected merging position output by the ramp intelligent agent is used to update the motion state of the main line intelligent agent; and the main line information fed back by the main line intelligent agent is used to update the motion state of the ramp intelligent agent. The application can improve the efficiency and safety of ramp CAV merging into the main line and guarantee the driving comfort of the main line vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a vehicle control method for main line multi-lane ramp merging area based on deep reinforcement learning. BACKGROUND

[0002] Due to the gradual popularization of autonomous driving technology and the widespread use of traditional vehicles, the mixed traffic flow formed by connected and autonomous vehicles (CAVs) and human-driven vehicles (HDVs) will continue to exist for a long time in the future. This transition period is due to differences in technology maturity, cost factors and user acceptance. During the ramp merging process on the highway, vehicles need to frequently adjust their operating states, such as speed changes, acceleration control and lane switching, to integrate into the main line traffic flow. This dynamic adjustment can easily lead to increased traffic flow instability, causing potential safety risks such as rear-end collisions and lane-changing conflicts. At the same time, the uncertainty and diversity of HDVs driving behavior, such as differences in driver reaction time, inconsistent decision-making and variable driving styles, seriously affect the efficiency and safety of CAVs merging into the main line on the ramp, which may lead to merging delays, traffic congestion and increased accident probability, thus challenging the reliability of the overall traffic system.

[0003] In the highly complex and dynamically changing scenario of ramp merging area, vehicle cooperative control faces significant challenges. Deep reinforcement learning (DRL) methods effectively deal with various environmental inputs, uncertainties and disturbances through continuous learning and adaptation during the training process, showing great potential. However, these methods still face challenges in computational efficiency and robustness when dealing with large-scale traffic flow, and there is a large modeling bias in mixed traffic flow scenarios. Therefore, multi-agent deep reinforcement learning (MADRL) has emerged, but due to the independent decision-making characteristics of each agent, strategy conflicts are easy to occur, and non-uniform distribution of reward functions may lead to local optimization goals deviating from global optimality. This makes existing MADRL methods still have problems such as strategy shock and slow convergence speed in mixed traffic flow cooperative control, and cannot efficiently and stably solve the vehicle merging decision control problem in the main line multi-lane ramp merging area under mixed traffic flow. SUMMARY

[0004] In view of the above problems in the prior art, the present application provides a main line multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning, which inputs CAVs as agents into a heterogeneous agent collaborative control framework, respectively adjusts the motion state of the main line multi-agent based on a multi-agent advantage actor-critic method with stable policy gradient, dynamically updates the motion state of the ramp agent based on a single-agent flexible actor-critic method exploring a continuous action space, so as to obtain a merging collaborative action, until each agent completes merging collaboration and drives off the main line multi-lane highway ramp merging area.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0006] A main line multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning, comprising the following steps:

[0007] Obtaining the motion state of the main line agent and the ramp agent in the main line multi-lane highway ramp merging area under mixed traffic flow;

[0008] Adjusting the motion state of the main line multi-agent in real time based on a multi-agent advantage actor-critic method with stable policy gradient;

[0009] Dynamically updating the motion state of the ramp agent based on a single-agent flexible actor-critic method exploring a continuous action space;

[0010] Building a heterogeneous agent collaborative control framework with the main line agent and the ramp agent, updating the motion state of the main line agent with the expected merging position output by the ramp agent, updating the motion state of the ramp agent with the main line information fed back by the main line agent, until each agent completes merging collaboration and drives off the main line multi-lane highway ramp merging area.

[0011] In some embodiments, the motion state of the main line agent and the ramp agent includes the state space, the action space and the reward and punishment function corresponding to each agent.

[0012] In some embodiments, the state space corresponding to the main line agent includes the current vehicle, the two vehicles closest to the current vehicle in front and behind in the vehicle perception range of the current lane, the two vehicles closest to the current vehicle in the adjacent lane in the vehicle perception range, and the distance from the current vehicle to the expected merging position of the ramp vehicle.

[0013] The state space corresponding to the ramp agent includes the current vehicle, the two vehicles closest to the current vehicle in the vehicle perception range, and the two vehicles closest to the expected merging point of the current vehicle on the outer lane of the main line in the vehicle perception range.

[0014] In some embodiments, the action space corresponding to the mainline agent comprises a vehicle acceleration and a vehicle lane-changing probability value;

[0015] The action space corresponding to the ramp agent comprises a vehicle acceleration, a vehicle lane-changing probability value, and an expected merging position of the ramp vehicle obtained according to the current state space.

[0016] In some embodiments, the reward and punishment function corresponding to the mainline agent comprises a weighted sum of a safety reward and punishment function of the agent, an efficiency reward and punishment function of the mainline agent, and a comfort degree reward and punishment function of the mainline agent;

[0017] The reward and punishment function corresponding to the ramp agent comprises a weighted sum of a safety reward and punishment function of the agent, an efficiency reward and punishment function of the ramp agent, a comfort degree reward and punishment function of the ramp agent, and a lane-changing reward and punishment function of the ramp agent.

[0018] In some embodiments, the safety reward and punishment function of the agent is specifically:

[0019] ;

[0020] wherein, is the safety reward and punishment function of the agent, is a distance of a preceding vehicle at a current time step, is a safe following distance, is a correlation coefficient.

[0021] In some embodiments, the efficiency reward and punishment functions of the mainline agent and the ramp agent are specifically:

[0022] ;

[0023] ;

[0024] wherein, is the efficiency reward and punishment function of the mainline agent, are all correlation coefficients, is a vehicle speed of the vehicle at a current time step, is a target vehicle speed of the vehicle on different roads, is a vehicle speed of the vehicle at a next time step, is the efficiency reward and punishment function of the ramp agent.

[0025] In some embodiments, the comfort degree reward and punishment functions of the mainline agent and the ramp agent are specifically:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] wherein, is a comfort reward-punishment function of the mainline agent, is a reward-punishment function of the mainline agent considering acceleration, is a reward-punishment function of the mainline agent considering jerk, is a comfort reward-punishment function of the ramp agent, is the acceleration of the vehicle at the current time step, are all correlation coefficients, is the acceleration of the vehicle at the next time step.

[0031] In some embodiments, the lane-changing reward-punishment function of the ramp agent is specifically:

[0032] ;

[0033] ;

[0034] ;

[0035] wherein, is a lane-changing reward-punishment function of the ramp agent, is a reward-punishment for the lane-changing position, is a reward for the success of lane-changing, is the lane-changing abscissa, is a correlation coefficient.

[0036] In some embodiments, the multi-agent advantage actor-critic method based on policy gradient stability adjusts the motion state of the mainline indefinite number of agents in real time, comprising:

[0037] The central value network is used to evaluate the influence of the action of each agent of the mainline on the state of all vehicles of the mainline, to obtain a central value reward, which is specifically:

[0038] ;

[0039] ;

[0040] ;

[0041] wherein, is a central value reward, is a reward-punishment for global collision, is a reward-punishment for global efficiency, is a safety coefficient, is an efficiency coefficient;

[0042] And the center value reward update strategy network is utilized to adjust the motion state of the main line indefinite quantity multi-agent in real time.

[0043] The present application has the following beneficial effects:

[0044] (1) In the cooperative framework of heterogeneous intelligent agents, the present application combines the policy gradient stability of multi-agent and the continuous action space explorability of single agent, fully explores the ramp merging action and the main line cooperative action, improves the convergence effect of the control method, improves the efficiency and safety of the ramp CAV merging into the main line, and guarantees the driving comfort of the main line vehicles.

[0045] (2) In the scene of the main line multi-lane highway ramp merging area under the mixed traffic flow, the vehicle merging decision control problem is modeled as a Markov process, and the corresponding state space, action space and reward mechanism are designed according to the driving target of the vehicles on different lanes, which can significantly improve the decision adaptability of CAV in dynamic complex scenes, optimize the traffic flow merging problem, and improve the intelligent level. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flow chart of the main line multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning is provided for the present application;

[0047] Figure 2 A schematic diagram of the main line multi-lane highway ramp merging area under mixed traffic flow is provided for the embodiment of the present application;

[0048] Figure 3 A schematic diagram of the cooperative framework of heterogeneous intelligent agents is provided for the embodiment of the present application;

[0049] Figure 4 A schematic diagram of the highway ramp entrance merging scene is provided for the embodiment of the present application;

[0050] Figure 5 A schematic diagram of the method update process is provided for the present application;

[0051] Figure 6 A comparison chart of the average reward of the present application and other methods is provided for the present application;

[0052] Figure 7 A comparison chart of the average speed of the ramp of the present application and other methods is provided for the present application;

[0053] Figure 8 A comparison chart of the average speed of the main line of the present application and other methods is provided for the present application;

[0054] Figure 9 A comparison chart of the average acceleration of the ramp of the present application and other methods is provided for the present application;

[0055] Figure 10Figure 8(a) is a plot of average speed for the mainline for the method of the present invention compared to other methods;

[0056] Figure 11 Figure 8(b) is a plot of merge time for the method of the present invention compared to other methods;

[0057] Figure 12(a) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a mainline of 800 veh / h;

[0058] Figure 12(b) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a mainline of 1600 veh / h;

[0059] Figure 12(c) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a ramp of 400 veh / h;

[0060] Figure 12(d) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a ramp of 800 veh / h;

[0061] Figure 12(e) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a CAV penetration of 20%;

[0062] Figure 12(f) is a plot of ramp vehicle speed for the method of the present invention compared to other methods for a CAV penetration of 80%;

[0063] Figure 13(a) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a mainline of 800 veh / h;

[0064] Figure 13(b) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a mainline of 1600 veh / h;

[0065] Figure 13(c) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a ramp of 400 veh / h;

[0066] Figure 13(d) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a ramp of 800 veh / h;

[0067] Figure 13(e) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a CAV penetration of 20%;

[0068] Figure 13(f) is a plot of ramp vehicle trajectory for the method of the present invention compared to other methods for a CAV penetration of 80%. DETAILED DESCRIPTION

[0069] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, all the inventions utilizing the concept of the present application are within the scope of the present application as long as various changes are obvious within the spirit and scope of the present application defined and determined by the appended claims.

[0070] As Figure 1 shown, the embodiment of the present application provides a main line multi-lane ramp merging area vehicle control method based on deep reinforcement learning, which comprises the following steps S1 to S4:

[0071] S1, obtaining the motion state of the main line agent and the ramp agent of the main line multi-lane ramp merging area under mixed traffic flow;

[0072] In the present embodiment, step S1 constructs a simulation scene of the main line multi-lane ramp merging area under mixed traffic flow based on the SUMO simulation platform, as shown in Figure 2 , a merging scene of main line double lanes and ramp single lane is constructed, which is more common in real scenes, and at the same time, lateral lane changing control and longitudinal speed control need to be considered comprehensively, and the merging control complexity is higher.

[0073] Based on the simulation scene of the main line multi-lane ramp merging area under mixed traffic flow, the motion state of the main line agent and the ramp agent is obtained. The motion state of the main line agent and the ramp agent includes the state space S, the action space A and the reward and punishment function R corresponding to each agent, as shown in Figure 3 . Wherein the main line multi-lane ramp merging area is main line double lane and ramp single lane.

[0074] The state space corresponding to the main line agent includes: the current vehicle, the front and rear two vehicles closest to the current vehicle in the vehicle perception range of the current lane, the front and rear two vehicles closest to the current vehicle in the vehicle perception range of the adjacent lane, and the distance from the current vehicle to the ramp vehicle expected merging point, specifically:

[0075] ;

[0076] Wherein: is the distance from the main line vehicle i to the ramp vehicle merging point, is the state space of the main line vehicle i and the ramp vehicle itself, is the state space of the front vehicle of the main line vehicle i, is the state space of the rear vehicle of the main line vehicle i, is the front vehicle state space of the left lane of the main line vehicle, is the rear vehicle state space of the left lane of the main line vehicle.

[0077] State space of ramp agent Including: the current vehicle, the two vehicles in front and behind the current vehicle within the vehicle perception range, and the two vehicles on the outside lane of the main line closest to the expected merging point of the current vehicle within the vehicle perception range; Specifically:

[0078] ;

[0079] Wherein: State space of ramp vehicle itself; State space of vehicle in front of ramp vehicle; State space of vehicle behind ramp vehicle; , The vehicle closest to the merging point of the ramp vehicle among the vehicles on the outside lane of the main line.

[0080] The state space of each vehicle includes four state characteristics, i.e. , , , . Among them The speed of vehicle i, The acceleration or deceleration of vehicle i, The horizontal coordinate of vehicle i, The vertical coordinate of vehicle i.

[0081] If there is no vehicle within the vehicle perception range, the state space is a set value.

[0082] Action space of main line agent Action space of ramp agent Including:

[0083] ;

[0084] ;

[0085] Wherein: Vehicle acceleration (m / s 2 ), positive value represents acceleration, negative value represents deceleration, 0 is to maintain the speed unchanged, the range is (-6, 6); Vehicle lane changing probability value, when the probability value is greater than 0.5, it can try to change lane; The expected merging position of ramp vehicle according to the current state space.

[0086] Reward and punishment mechanism of main line agent Reward and punishment mechanism of ramp agent Including:

[0087] ;

[0088] ;

[0089] wherein: , , is a main line agent reward function coefficient, , , , is a ramp agent reward function coefficient, is a safety reward and punishment mechanism of the main line agent and the ramp agent, is an efficiency reward and punishment mechanism of the main line agent, is a comfort reward and punishment mechanism of the main line agent, is an efficiency reward and punishment mechanism of the ramp agent, is a comfort reward and punishment mechanism of the ramp agent, is a lane changing reward and punishment mechanism of the ramp agent.

[0090] In the embodiment, the main line agent reward function coefficient is 2, the ramp agent reward function coefficient is 1.2, the safety reward and punishment mechanism of the main line agent and the ramp agent is 1, the efficiency reward and punishment mechanism of the main line agent is 1.8, the comfort reward and punishment mechanism of the main line agent is 2.3, the efficiency reward and punishment mechanism of the ramp agent is 6.8, and the comfort reward and punishment mechanism of the ramp agent is 1.2. In the safety reward and punishment mechanism of the vehicle, the vehicle predicted time to collision (TTC) is used to describe the safety in the vehicle following driving, and the calculation formula is as follows:

[0091] ;

[0092] wherein: is a position difference between two vehicles; is a speed difference between two vehicles; is a vehicle length.

[0093] The safety reward is calculated by taking the TTC as a limit collision parameter, the expected driving distance under the current vehicle speed condition is calculated, and the agent is guided to keep following driving at a safe following distance by comparing with the actual distance of the front vehicle. The safety reward and punishment mechanism of each agent of the main line and the ramp agent as follows:

[0094] ;

[0095] wherein, is a safety reward and punishment function of the agent, is a distance of a front vehicle at a current time step, is a safe following distance, ;​​​​​​ is a correlation coefficient.

[0096] In the vehicle efficiency reward and punishment mechanism, in order to reduce the main line speed interference and improve the ramp vehicle lane changing success rate, the speed reward and punishment of the main line and ramp agent is needed to guide the ramp vehicle to drive at a speed close to the target speed and the main line vehicle to drive at a higher speed as much as possible. The efficiency reward and punishment of each agent on the main line and the ramp agent is , :

[0097] ;

[0098] ;

[0099] wherein, is the efficiency reward and punishment function of the main line agent, are correlation coefficients, is the speed of the vehicle at the current time step, is the target speed of the vehicle on different roads, the ramp is 16.67 m / s, and the main line is 27.78 m / s; is the speed of the vehicle at the next time step, is the efficiency reward and punishment function of the ramp agent.

[0100] During the driving of the vehicle, the ramp vehicle continuously obtains the current lane speed limit and updates its own reward function. When the speed is continuously increased, the reward is also continuously increased. The main line vehicle encourages driving close to the target speed and the speed change of the front and rear vehicles is small.

[0101] In the vehicle comfort reward and punishment mechanism, when the ramp vehicle merges, if there is a gap but the current speed does not support it to complete the merging, some vehicles may adopt aggressive driving strategies, which affects the comfort of passengers, increases the risk of traffic accidents, and destroys the stability of the main line traffic flow.

[0102] Therefore, the aggressive driving strategy is given a negative comfort penalty, and the comfort reward and punishment of the main line agent and the ramp agent is , . This penalty can be achieved by increasing the negative consequences of these behaviors, thereby suppressing such behaviors. Specifically,

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] wherein, a comfort reward function for the mainline agent, a reward function for the mainline agent considering acceleration, a reward function for the mainline agent considering jerk, a comfort reward function for the ramp agent, an acceleration of the vehicle at the current time step, are all correlation coefficients, an acceleration of the vehicle at the next time step.

[0108] In the ramp vehicle lane-changing reward-punishment mechanism, since the ramp vehicle may face forced lane-changing at the end of the acceleration lane during the merging process, the lane-changing position reward-punishment is designed for the ramp vehicle to encourage the ramp agent to complete lane-changing early. Specifically,

[0109] ;

[0110] ;

[0111] ;

[0112] wherein, is a lane-changing reward-punishment function for the ramp agent, is a reward-punishment for the lane-changing position, is a reward for the success of lane-changing, is a lane-changing abscissa, is a correlation coefficient.

[0113] S2, based on the policy gradient stable multi-agent advantage actor-critic method, the motion state of the mainline indefinite number of agents is adjusted in real time;

[0114] In the embodiment, step S2 adopts a multi-agent advantage actor-critic method (MAA2C) based on policy gradient stability to adjust the motion state of the mainline indefinite number of agents in real time, including: the mainline indefinite number of agents adjust the motion state according to the decision module and the update module.

[0115] The decision module includes a policy network, an action space, an individual reward, and a central value reward, and the update module includes a value network and a global experience pool.

[0116] The value network includes an individual value network and a central value network.

[0117] As Figure 3 ​As shown, the main line agent inputs its own motion state (including itself, surrounding vehicles and merging point distance) into its policy network, and the policy network outputs the respective action (acceleration and lane changing probability). At the same time, the states and actions of all main line vehicles are sent to the central value network for calculating the central value reward and punishment for evaluating the overall strategy, so as to guide the strategy update in the direction conducive to group cooperation. The main line agent stores the experience (state, action, reward, new state) of this interaction into the global experience pool to provide data for multi-agent common learning; in the update process, the central value network samples a batch of experience data from the global experience pool to calculate the central value reward and punishment, and the main line agent samples a batch of experience data from the global experience pool to calculate the value loss of each agent using the individual value network to update the critic network of itself. The advantage function is calculated using the updated critic network and the central value reward and punishment to guide the update of the policy network of the main line vehicle, with the goal of maximizing long-term returns while taking into account the global collaboration evaluated by the central value network.

[0118] In the central value network reward and punishment mechanism of the main line agent, the central value network is responsible for evaluating the influence of the action of each agent on the state of all vehicles in the main line, thereby reducing the acceleration fluctuation and collision conflict of the main line vehicles and improving the traffic stability of the main line. Therefore, the global reward and punishment is designed to take into account both safety and efficiency :

[0119] ;

[0120] ;

[0121] ;

[0122] wherein, is the central value reward and punishment, is the reward and punishment for global collision, is the global efficiency reward and punishment, is the safety coefficient, is the efficiency coefficient, which is adjusted according to the real-time traffic density to ensure the balance between safety and efficiency.

[0123] S3, a single-agent flexible actor-critic method based on continuous action space exploration dynamically updates the motion state of the ramp agent;

[0124] In this embodiment, step S3 employs a single-agent flexible actor-critic method based on continuous action space exploration to dynamically update the motion state of the ramp agent, which includes: the ramp agent adjusts the motion state according to the decision module and the update module;

[0125] The decision module includes a policy network, an action space and a reward and punishment mechanism, and the update module includes a value network and an experience pool.

[0126] The value network includes two value networks and two target value networks.

[0127] As shown in Figure 3 The ramp agent inputs its own motion state (including itself, the vehicles before and after the ramp, and the two vehicles closest to the ramp merging point on the main line) into its policy network. The policy network outputs actions (acceleration, lane changing probability, and expected merging position). The ramp agent stores the experience of this interaction (state, action, reward, new state) in the experience pool, samples data from the experience pool, and uses two value networks and two target value networks for calculation and updating to reduce the bias of value estimation and improve training stability. The goal of updating the policy network is to maximize the expected return while maximizing the entropy of the policy, thereby encouraging exploration of more possible successful merging strategies and enhancing robustness.

[0128] S4, build a heterogeneous agent cooperative control framework with main line agents and ramp agents, update the motion state of the main line agent using the expected merging position output by the ramp agent, and update the motion state of the ramp agent using the main line information fed back by the main line agent, until each agent completes the merging cooperation and drives off the main line multi-lane highway ramp merging area.

[0129] As shown in Figure 3 The ramp agent outputs actions through the policy network according to its own state, which contains key planning information, i.e., the expected merging position. This expected merging position is sent to all relevant main line agents in real time. After each main line agent receives it, it will immediately calculate the distance to this merging point, so that the main line vehicles are no longer passive reactions, but can predict where the conflict will occur. The state information of the main line agent is fed back to the ramp agent in real time. According to the state space of the ramp agent, the system will filter the main line vehicle state information and feed it back to the ramp agent as input to its state space.

[0130] To explain the method of the present application, the method of the present application is verified and analyzed below in combination with an implementation case. The present application uses SUMO and Python for joint development to verify the main line multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning of the present application.

[0131] wherein SUMO (Simulation of Urban MObility) is an open-source, highly portable, multi-modal traffic simulation tool suitable for large-scale traffic networks, allowing accurate modeling of complex traffic flow in freeway on-ramp merge areas and road networks. Deep reinforcement learning is a data-driven approach that learns optimal strategies autonomously by interacting with the environment and receiving feedback signals. In the present invention, the agent takes vehicle states, road information, and traffic flow parameters as input and outputs speed control instructions to gradually optimize the efficiency and safety of the merging process. First, the present invention is compared with other methods through simulation, and the parameters used are shown in Table 1:

[0132] Table 1 Simulation parameter table

[0133]

[0134] A freeway on-ramp merge scenario is set up in SUMO as shown in Figure 4 , which includes a two-lane mainline and a single-lane ramp. The length of the merging area is set to 150m, the start of merging is at 250m, and the end of merging is at 400m. The target speed of the mainline is 27.78m / s, and the target speed of the ramp is 16.67m / s. Vehicles arrive randomly.

[0135] The training process of the present invention and other methods is carried out in a traffic scenario with 1200 veh / h on the mainline, 600 veh / h on the ramp, and a CAV penetration rate of 50%. During the training process, without updating the network, 10% of the training period is run to allow the agent to gain enough experience. With a training period of 60s, if the ramp vehicle does not complete merging within this time range, the merging is considered a failure. In each training period, the first 20s is used for environment warming to ensure that the simulation road network is full of vehicles.

[0136] Subsequently, the agent collects state transition data through real-time interaction with the SUMO simulation environment, and updates the network parameters at the end of each period. The internal update process of the present invention is shown in Figure 5 .

[0137] Based on the above experimental environment and experimental method, the present invention is superior to other deep reinforcement learning methods in terms of traffic efficiency, smooth comfort, and merging efficiency. Figure 6The average reward value obtained by the MAA2C-SAC method in the training period is significantly higher than that of MADDPG and MADQN. It is worth noting that MAA2C-SAC reaches a stable convergence state in the middle of training (about 700 periods), while MADQN does not start to converge until 1800 periods, which shows that the central value network and reward mechanism effectively alleviate the problem of unstable strategy in multi-agent training. At the same time, MADDPG converges to a local optimum in the early training period and does not explore the global optimal strategy.

[0138] Figure 7 and Figure 9 The joint comparison data show that the average speed of ramp vehicles controlled by MAA2C-SAC is increased by 71.09% and 18.16% compared with MADQN and MADDPG respectively, and the lower average acceleration of the ramp vehicles also makes the ramp vehicles have better driving comfort during the fast merging process. Among them, MADDPG excessively pursues to increase the speed, which leads to a decrease in the merging time window, resulting in a situation of waiting for merging at the merging end.

[0139] Figure 8 and Figure 10 The joint comparison data show that the MAA2C-SAC method successfully maintains the average speed of the main line vehicles in the range of 26 m / s under the premise of ensuring safety, close to the target speed. While the other two methods excessively pursue high speed and high acceleration on the main line, resulting in the failure of cooperative ramp merging. At the same time Figure 9 shows that the MAA2C-SAC method can maintain the driving comfort of the main line vehicles (the average acceleration is around -0.5 m / s2~0.5 m / s2).

[0140] Figure 11 The merging time distribution curves of

[0141] To verify the generalization ability of the method in complex traffic flow, further multi-scenario simulation tests are carried out. The experiments cover different main line vehicle flows, different ramp vehicle flows and different CAV penetration rates, and the strategy is verified and analyzed by comparing the uncontrolled and controlled. The results show that MAA2C-SAC performs well in various scenarios, especially in high traffic density and low CAV penetration rate conditions, and can still maintain the merging efficiency and safety.

[0142] As shown in Figures 12(a) to 12(f) , it is shown that under different traffic flows, the MAA2C-SAC method can maintain the stable acceleration of the ramp vehicles, and accelerate to the main line speed range as much as possible before the merging point appears, while maintaining a higher passing efficiency.Figures 12(a) to 12(f) It is shown that even in the traffic scene with low CAV penetration rate, the merging vehicle can maintain stable speed growth to complete the merging task by relying on the dynamic learning ability of the built-in SAC system. By comparing the uncontrolled scene, the merging vehicle is prone to sudden acceleration behavior, and when the main line traffic flow density is at a high level, the vehicle may trigger the emergency braking deceleration mechanism due to insufficient lane changing space. In summary, the method of the application can effectively suppress the violent fluctuation phenomenon of the speed field in the merging area, and achieve the control goal of fast and smooth merging.

[0143] As shown in the merging trajectory diagram Figures 13(a) to 13(f) illustrates the effectiveness of the MAA2C-SAC method in various traffic scenes. Figures 13(a) to 13(f) It is shown that when the ramp vehicle is preparing to change lanes, the upstream vehicle can change lanes to the inner lane in advance to create a safe merging gap. At the same time, the main line vehicle has a small range of trajectory fluctuation during the merging process, i.e., less sudden acceleration and deceleration behavior. In summary, under the control of MAA2C-SAC, the main line vehicle can complete the merging with the ramp vehicle while ensuring normal driving within a small influence range, reducing the speed disturbance of the main line, and improving the safety and stability of the merging area.

[0144] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate one or more means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0145] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more blocks or any combination thereof.

[0147] The principles and implementations of the present application are described in the specific examples. The above examples are used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

[0148] Those skilled in the art will realize that the examples described herein are for the purpose of aiding the reader in understanding the principles of the present application and should be understood as not limiting the scope of protection of the present application. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A mainline multi-lane highway ramp merging area vehicle control method based on deep reinforcement learning, characterized in that, The method comprises the following steps: obtaining the motion state of the main line agent and the ramp agent in the main line multi-lane ramp merging area under mixed traffic flow; wherein the motion state of the main line agent and the ramp agent comprises a state space, an action space and a reward and punishment function corresponding to each agent; the reward and punishment function corresponding to the main line agent comprises a weighted sum of the safety reward and punishment function of the agent, the efficiency reward and punishment function of the main line agent and the comfort reward and punishment function of the main line agent; the reward and punishment function corresponding to the ramp agent comprises a weighted sum of the safety reward and punishment function of the agent, the efficiency reward and punishment function of the ramp agent, the comfort reward and punishment function of the ramp agent and the lane changing reward and punishment function of the ramp agent; The safety reward and punishment function of the agent is specifically: ； wherein, is a safety reward-punishment function of the agent, is a distance of the preceding vehicle at the current time step, is a safe following distance, is a correlation coefficient; The efficiency reward and punishment function of the main line agent and the ramp agent is specifically: ； ； wherein, is the efficiency reward-penalty function of the mainline agent, are correlation coefficients, is the speed of the vehicle at the current time step, is the target speed of the vehicle on different roads, is the speed of the vehicle at the next time step, is the efficiency reward-penalty function of the ramp agent; The comfort reward and punishment function of the main line agent and the ramp agent is specifically: ； ； ； ； wherein, a comfort reward function for the mainline agent, a reward function for the mainline agent considering acceleration, a reward function for the mainline agent considering jerk, a comfort reward function for the ramp agent, an acceleration of the vehicle at the current time step, are all correlation coefficients, an acceleration of the vehicle at the next time step; The lane changing reward and punishment function of the ramp agent is specifically: ； ； ； wherein, is a reward-penalty function for ramp metering agents, is a reward-penalty for ramp position, is a reward for ramp success, is a ramp x-coordinate, is a correlation coefficient; The multi-agent advantage actor-critic method based on policy gradient stability adjusts the motion state of the main line agent in real time; The single-agent flexible actor-critic method based on continuous action space exploration dynamically updates the motion state of the ramp agent; A heterogeneous agent cooperative control framework is constructed by the main line agent and the ramp agent, the expected merging position output by the ramp agent is used to update the motion state of the main line agent, and the main line information fed back by the main line agent is used to update the motion state of the ramp agent, until each agent completes the merging cooperation and drives off the main line multi-lane ramp merging area.

2. The deep reinforcement learning based mainline multilane highway ramp-merging area vehicle control method of claim 1, wherein, The state space corresponding to the main line agent comprises the current vehicle, the two vehicles closest to the current vehicle in front and behind in the vehicle perception range of the current lane, the two vehicles closest to the current vehicle in the adjacent lane in the vehicle perception range and the distance from the current vehicle to the expected merging position of the ramp vehicle; The state space corresponding to the ramp agent comprises the current vehicle, the two vehicles closest to the current vehicle in front and behind in the vehicle perception range and the two vehicles closest to the expected merging point of the current vehicle on the outer lane of the main line in the vehicle perception range. 3.The deep reinforcement learning based mainline multi-lane freeway ramp-merging area vehicle control method of claim 1, wherein, The action space corresponding to the main line agent comprises vehicle acceleration and vehicle lane changing probability value; The action space corresponding to the ramp agent comprises vehicle acceleration, vehicle lane changing probability value and the expected merging position obtained by the ramp vehicle according to the current state space. 4.The deep reinforcement learning based mainline multi-lane highway ramp-merging area vehicle control method of claim 1, wherein, The multi-agent advantage actor-critic method based on policy gradient stability adjusts the motion state of the main line agent in real time comprises: The central value network is used to evaluate the influence of the action of each agent on the state of all vehicles on the main line to obtain a central value reward, which is specifically: ； ； ； wherein, a central value award, a global collision penalty, a global efficiency award, a safety factor, an efficiency factor; And the central value reward is used to update the policy network to adjust the motion state of the main line agent in real time.

Images