Method, system and equipment for controlling traffic of vehicles at non-signalized intersection

Through the combination of the MADDPG algorithm and the follow-up model IDM, an adaptive fleet scale and acceleration strategy are generated, which solves the efficiency and safety of vehicle formations without signal intersections in a dynamic environment, and achieves more efficient vehicle traffic control.

CN119068660AActive Publication Date: 2024-12-03CHANGAN UNIV

Patent Information

Application Number
CN202411114593.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-03
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

In a dynamically changing traffic environment, the existing vehicle fleet algorithm without signal intersections may lead to unreasonable fleet size, causing unnecessary slowing or parking, and affecting traffic efficiency.

Method used

Multi-agent confrontation deep reinforcement learning (MADDPG) algorithm is used to generate vehicle adaptive formation strategies suitable for continuous action space through actor network and critical network, optimize fleet size and head vehicle acceleration, and combine the following model IDM for fleet coordination control. Ensure the safety, traffic efficiency and driving comfort of the fleet within signal-free intersections.

Benefits of technology

Generate a reasonable fleet size in a dynamic traffic environment, avoid unnecessary slowing or parking, improve the traffic efficiency of vehicles without signal intersections, and improve traffic safety and driving comfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068660A_ABST
    Figure CN119068660A_ABST
Patent Text Reader

Abstract

The invention discloses a non-signalized intersection vehicle passing control method, system and device, and relates to the technical field of non-signalized intersection passing, and the method comprises the steps: obtaining the observation states of a target vehicle team in a non-signalized intersection environment and surrounding vehicle teams; the observation states of the target motorcade and the surrounding motorcade are input into an actor network, actions of the target motorcade are obtained, and the actions comprise the scale of the target motorcade and the acceleration of the head vehicle of the target motorcade; inputting the observation state and action of the target motorcade into a critic network to obtain an estimated Q value; the Q value is evaluated, the evaluation result is fed back to the actor network to be combined with the total reward, the strategy of the optimal motorcade scale and the optimal motorcade head vehicle acceleration is obtained through continuous updating, and therefore coordination control is conducted on the target motorcade; according to the method, in a dynamically changing traffic environment, vehicle adaptive formation and vehicle team coordination control can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control at unsignalized intersections, and in particular to a method, system and device for controlling traffic at unsignalized intersections. Background Art

[0002] Unsignalized intersections are an important part of the urban traffic system, and their traffic efficiency directly affects the traffic flow of the entire city. However, traditional unsignalized intersection management methods often rely on traffic control facilities or fixed traffic rules. Although a large number of researchers have optimized the traffic of vehicles at unsignalized intersections based on traffic rules and proposed a series of rules and protocols that are superior to signal control methods, these rules and protocols are difficult to adapt to complex and changing traffic environments and improve the traffic efficiency of intersections. In order to optimize traffic performance, researchers began to establish optimization models and solve the optimal solution to obtain unsignalized intersection control solutions. However, with the increase in the number of vehicles to be planned, the number of effective traffic sequence combinations has exploded, resulting in an increase in the amount of calculation required to solve the problem, making it difficult to solve the optimization model and unable to meet the timeliness requirements of real-time traffic control problems. Deep reinforcement learning (DRL), as an artificial intelligence method that combines deep learning and reinforcement learning, has powerful perception and decision-making capabilities. In the DRL framework, each vehicle is regarded as an independent intelligent agent that adapts to changing environments and handles complex decision-making tasks through continuous trial and error and learning. Using DRL to design a traffic strategy that can adapt to changes in the traffic environment of unsignalized intersections has become the focus of current research. However, this method requires a large amount of data and computing resources for training. As the number of agents increases, the time and difficulty of training also increase significantly, which directly affects the performance of the model.

[0003] Vehicle platooning can achieve the goals of reducing communication burden, improving traffic efficiency and reducing energy consumption by platooning similar vehicles. In addition, after the vehicles are platooned, multiple vehicles can be controlled as one vehicle, thereby reducing the control overhead. Applying vehicle platooning to the multi-agent deep reinforcement learning (MADRL) algorithm can reduce the number of agents, reduce the difficulty of training, and optimize the training performance. In the existing research on vehicle platooning at unsignalized intersections, most studies tend to fix the formation, that is, once the formation is formed, it will maintain a fixed formation size and cooperate as a whole during the entire driving process.

[0004] The realization of this formation is stable and reliable, but in a dynamically changing traffic environment, the algorithm may lead to an unreasonable fleet size, causing unnecessary deceleration or parking, and affecting traffic efficiency. A small number of studies have used DRL-based adaptive formations to determine the optimal fleet size, but these studies use less efficient rules or optimization methods when coordinating fleets. There are also studies that have achieved adaptive formations of vehicles and efficient coordination of fleets by introducing deep Q networks (DQN). The fleet coordination actions generated by the adaptive formation traffic strategy using the DQN algorithm are discrete, while the fleet driving actions are continuous. For the continuous action space, the discretization processing of DQN will lead to a decrease in control accuracy, affecting traffic safety and traffic efficiency, and also ignoring the performance of driving comfort. Summary of the invention

[0005] In view of the shortcomings of the prior art that in a dynamically changing traffic environment, the size of the vehicle fleet may be unreasonable, causing unnecessary deceleration or parking, and affecting the traffic efficiency, the present invention proposes a vehicle traffic control method, system and equipment for an unsignalized intersection. By adopting the MADDPG algorithm suitable for continuous action space problems, with safety, traffic efficiency and driving comfort as the goals, vehicle adaptive platooning and fleet coordination control are performed, thereby solving the problems existing in the prior art.

[0006] A method for controlling vehicle traffic at an unsignalized intersection comprises the following steps:

[0007] Acquire the current observation status of the target fleet and the decision reference fleet in the unsignalized intersection environment; the observation status includes the position, size, acceleration and lane position of the target fleet and the decision reference fleet; the decision reference fleet represents a fleet that has a conflicting relationship with the lane position of the target fleet;

[0008] The current observation states of the target fleet and the decision reference fleet are input into the actor network of the MADDPG algorithm to generate the action to be performed by the target fleet at the next moment; wherein the action includes adjusting the size of the target fleet and the acceleration of the lead vehicle of the target fleet; according to the adjusted size of the target fleet and the acceleration of its lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet;

[0009] The current observation status of the target fleet and the decision reference fleet, as well as the action to be performed by the target fleet at the next moment, are input into the critic network to obtain an estimated Q value; the action to be performed by the target fleet at the next moment is evaluated based on the Q value, and the evaluation result is fed back to the actor network and combined with the total reward, the action to be performed by the target fleet at the next moment is continuously updated and optimized iteratively, and a strategy for adjusting the optimal fleet size and the optimal acceleration of the leading vehicle of the fleet is generated;

[0010] The target fleet in the unsignalized intersection environment is coordinated and controlled according to the strategy of adjusting the optimal fleet size and the optimal fleet lead vehicle acceleration of the target fleet, as well as the acceleration of the vehicles following the lead vehicle; wherein, within one control cycle, all fleets in the unsignalized intersection environment will successively become the target fleet to execute actions, thereby completing the traffic control of all fleets in the unsignalized intersection.

[0011] Furthermore, the acquisition of the current observation state of the target fleet and the decision reference fleet in the unsignalized intersection environment includes an observation state for generating the size of the target fleet and an observation state for generating the acceleration of the leading vehicle of the target fleet; the observation state for generating the size of the target fleet include:

[0012] The state of the target fleet at time t is

[0013] The decision reference fleet state is expressed as:

[0014]

[0015] in, is the position of team i, is the size of fleet i, is the speed of the leading vehicle in fleet i, is the lane number where team i is located, is the position of the nth decision reference fleet around fleet i, is the size of the nth decision reference fleet around fleet i, is the speed of the leading vehicle of the nth decision reference fleet around fleet i, The lane number of the nth decision reference fleet around fleet i;

[0016] The observation state used to generate the acceleration of the lead vehicle in the target convoy include:

[0017] The state of the target fleet at time t is expressed as:

[0018] The decision reference fleet state is expressed as:

[0019]

[0020] in, is the acceleration of the leading vehicle in convoy i, is the acceleration of the leading vehicle in the nth decision reference convoy around convoy i.

[0021] Furthermore, the total reward includes a total reward for generating an optimal fleet size strategy with fairness, traffic efficiency and safety as optimization objectives, and a total reward for generating an optimal fleet lead vehicle acceleration strategy with safety, traffic efficiency and driving comfort as optimization objectives.

[0022] Furthermore, the total reward for generating the optimal fleet size strategy with fairness, traffic efficiency and safety as optimization goals is It is expressed as:

[0023]

[0024] Among them, ω pf Reward for fairness The weight, ω pe Indicates traffic efficiency reward The weight, ω ps Safety Reward The fairness reward is measured by the time each team stays in the control area. The longer the team stays in the control area, the smaller the fairness reward. It is expressed as:

[0025]

[0026] in: is the total time step that the first vehicle in the i-th fleet stays, is the size of the i-th fleet, w thr is the waiting time threshold, c is the control Parameters of the upper limit; represents the length of time the i-th convoy stays in the control area;

[0027] The efficiency bonus is expressed as the average speed of the fleet With maximum speed v M The closer the speed is to the maximum speed, the higher the traffic efficiency is, and the greater the traffic efficiency reward fed back to the fleet is. It is expressed as:

[0028]

[0029] The safety reward is expressed as the ratio of the distance between the convoys to their corresponding safety distance. The smaller the spatial or temporal distance, the higher the risk of collision between the convoys, and the smaller the safety reward. It is expressed as:

[0030]

[0031] Among them, S d (t) and S t(t) are the spatial and temporal distances between the two convoys, d p_thr and t p_thr are the spatial and temporal safety distance thresholds of the two convoys respectively.

[0032] Furthermore, the total reward of the optimal acceleration strategy for the leading vehicle in the convoy is generated by taking safety, traffic efficiency and driving comfort as the optimization objectives. It is expressed as:

[0033]

[0034] If the team runs normally, the total reward It is the weighted sum of the rewards for each target. If the team collides, the total reward is -10;

[0035] The security reward r s (t), traffic efficiency reward r e (t) and driving comfort reward r j (t) are respectively expressed as:

[0036]

[0037] Among them, S d (t) and S t (t) are the spatial and temporal distances between the two convoys, (x i (t),y i (t)) represents the position of team i at time t, (x i-1 (t),y i-1 (t)) represents the position of team i-1 at time t, l i-1 is the length of the convoy i-1, and the convoy length is l = d last -d first It is calculated that d last Indicates the distance between the last vehicle in the convoy and the stop line, d first Indicates the distance between the first vehicle in the convoy and the stop line; (0,d p_thr ) represents the spatial distance range where there is a risk of collision, d p_thr =D thr +l i-1 , D thr is the safety distance threshold, v i (t) is the speed of team i at time t, t r is the delay time, including the driver's reaction time and the system's reaction time during braking, a max is the maximum braking acceleration; (0,t p_thr) represents the time distance range where there is a risk of collision, v M is the maximum speed limit at the intersection, v m is the minimum speed limit at the intersection; a t is the acceleration at the current moment, a t-1 is the acceleration at the previous moment, Δa max is the maximum value of acceleration change;

[0038] The judgment condition for the convoy to collide is:

[0039]

[0040] Among them, D p (t) is the fleet p 1 and the fleet 2 The distance between the two teams at time t; (x p1 (t),y p1 (t)) and (x p2 (t),y p2 (t)) is the fleet p 1 and the fleet 2 At the position at time t, l 1 and l 2 The fleet p 1 and the fleet 2 Length; d gap Represents the collision gap.

[0041] Furthermore, the action includes adjusting the size of the target convoy and the acceleration of the head vehicle of the target convoy; the action space of the size of the target convoy is expressed as:

[0042]

[0043] The actions of the team It is expressed as:

[0044]

[0045] Among them, θ l The actor network weights representing the size of the generated fleet, Indicates in status The action value calculated according to the strategy π;

[0046] The action space of the acceleration of the target convoy head vehicle is expressed as The action of team i at time step t It is expressed as:

[0047]

[0048] Among them, π is the actor network control strategy; θ i Represents the weights of the actor network.

[0049] Furthermore, according to the adjusted target fleet size and the acceleration of the lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet. It is expressed as:

[0050]

[0051] in, is the expected following distance of vehicle j in fleet i; s 0 is the minimum safety distance; v j is the current speed of the jth vehicle in the convoy i; T is the safe following distance; Δvx is the speed difference between the jth vehicle in the convoy i and the preceding vehicle; a max is the maximum acceleration; a c is the comfortable deceleration; δ is the sensitivity parameter of acceleration; is the following acceleration of the jth vehicle in the convoy i; v 0 is the initial velocity of the jth vehicle in the fleet i; Δs j is the distance between the jth vehicle and the vehicle in front.

[0052] Furthermore, the target convoy in the unsignalized intersection environment is coordinated and controlled according to the strategy of adjusting the optimal convoy size and the optimal convoy head vehicle acceleration of the target convoy, and the acceleration of the vehicle following the head vehicle, specifically including the following steps:

[0053] If the target fleet If the size of the fleet is smaller than the optimal fleet size, the fleet behind The vehicle information of and information;

[0054] If the target fleet If the size of the fleet is smaller than the optimal fleet size, the fleet behind The vehicle information of and information;

[0055] like If the size is greater than the optimal fleet size, The excess vehicles are separated and reorganized into a fleet;

[0056] The generated acceleration of the target vehicle and the acceleration of the following vehicle are used to control the passage of each vehicle in the convoy, which can be expressed as:

[0057]

[0058] in, represents the position of team i at time t, represents the position of team i at time t+Δt, Δt is the time of a control cycle, represents the acceleration of the ith team at time t, represents the speed of team i at time t, represents the speed of team i at time t+Δt.

[0059] The present invention also includes a vehicle traffic control system for an unsignalized intersection, comprising:

[0060] An acquisition module is used to acquire the current observation status of the target fleet and the decision reference fleet in the unsignalized intersection environment; the observation status includes the position, size, acceleration and lane position of the target fleet and the decision reference fleet; the decision reference fleet represents a fleet that has a conflicting relationship with the lane position of the target fleet;

[0061] The action calculation module is used to input the current observation states of the target fleet and the decision reference fleet into the actor network of the MADDPG algorithm to generate the action to be performed by the target fleet at the next moment; wherein the action includes adjusting the size of the target fleet and the acceleration of the lead vehicle of the target fleet; according to the adjusted size of the target fleet and the acceleration of its lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet;

[0062] The optimization module is used to input the current observation status of the target fleet and the decision reference fleet, as well as the action to be performed by the target fleet at the next moment, into the critic network to obtain an estimated Q value; the action to be performed by the target fleet at the next moment is evaluated according to the Q value, and the evaluation result is fed back to the actor network and combined with the total reward, and the action to be performed by the target fleet at the next moment is continuously updated and optimized iteratively, and a strategy for adjusting the optimal fleet size and the optimal acceleration of the leading vehicle of the fleet is generated;

[0063] The control module is used to coordinate the control of the target fleet in the unsignalized intersection environment according to the strategy of adjusting the optimal fleet size and the optimal fleet head vehicle acceleration of the target fleet, as well as the acceleration of the vehicle following the head vehicle; wherein, within a control cycle, all fleets in the unsignalized intersection environment will successively become target fleets to execute actions, thereby completing the traffic control of all fleets in the unsignalized intersection.

[0064] The present invention also includes a computer device for controlling vehicle traffic at an unsignalized intersection, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the method for controlling vehicle traffic at an unsignalized intersection when executing the computer program.

[0065] The present invention provides a method, system and device for controlling vehicle traffic at an unsignalized intersection, which have the following beneficial effects:

[0066] The present invention adopts the MADDPG algorithm suitable for continuous action space problems, obtains the size of the target fleet and the acceleration of the head vehicle of the target fleet in the actor network, obtains the Q value by inputting the observation state and action of the target fleet and the decision reference fleet into the critic network, and evaluates the strategy of the optimal fleet size and the optimal fleet head vehicle acceleration according to the Q value and the total reward, and performs vehicle adaptive formation and fleet coordination control according to the strategy and the acceleration of the following vehicles in the fleet; the method can generate a reasonable fleet size in a dynamically changing traffic environment, control the passage of all fleets in an unsignalized intersection, avoid unnecessary deceleration or parking, and improve the vehicle passage efficiency at the unsignalized intersection. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Flow chart of a method for controlling vehicle traffic at an unsignalized intersection according to an embodiment of the present invention;

[0068] Figure 2 A schematic diagram of the environment interaction mechanism and network structure in an embodiment of the present invention;

[0069] Figure 3 1 is a flowchart of the MADDPG algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0071] The present invention adopts the MADDPG algorithm suitable for continuous action space problems, takes fairness, traffic efficiency and safety as optimization goals, performs vehicle adaptive platooning, and takes safety, traffic efficiency and driving comfort as optimization goals, performs fleet coordination control, such as Figure 1 As shown, the specific steps include:

[0072] S1. Use MADDPG to generate the optimal size N of each fleet based on the state obtained from the current unsignalized intersection environment (in the initial state, each car is a fleet); MADDPG mainly includes elements such as state, action, multi-objective reward function and network.

[0073] The state consists of key information such as the position (p), size (N), speed (v) and lane number (lane) of the fleet. i When taking action at time t, we also need to consider the impact of the conflicting lanes on the agent. i By taking the state information of the target fleet and n decision reference fleets as the input of the network, the algorithm can calculate and understand the environment's demand for fleet size and provide a basis for generating decision actions. In the MADDPG algorithm for adaptive formation (generating target fleet size), the input state is the state of the target fleet and its surrounding n decision reference fleets. The state of the target fleet at time t is expressed as The states of the surrounding n decision reference fleets that affect the size of the fleet are expressed as follows:

[0074]

[0075] in, is the position of team i, is the size of fleet i, is the speed of the leading vehicle in fleet i, is the lane where the team i is located, is the position of the nth convoy around convoy i, is the size of the nth convoy around convoy i, is the speed of the leading vehicle in the nth convoy around convoy i, The lane number of the nth convoy around convoy i.

[0076] In the MADDPG algorithm for adaptive formation, the action is the size of the target fleet, which is calculated by the actor network based on the current state. In time step t, the actions of n+1 fleets (fleet i and its surrounding n decision reference fleets) in the current environment need to be calculated, and the action space is expressed as The actions of the team The calculation is as follows:

[0077]

[0078] Among them, θl represents the actor network weight that generates the fleet size, Indicates in status The action value calculated according to the strategy π.

[0079] In the MADDPG algorithm for adaptive platooning, the multi-objective reward comprehensively considers the three goals of fairness, efficiency, and safety. The fairness reward aims to ensure that each convoy has equal access to the intersection, and is measured by the time each convoy stays in the control zone. The longer the convoy stays in the control zone, the smaller its fairness reward. The calculation is as follows:

[0080]

[0081] in: is the total time step that the first vehicle in the i-th fleet stays, Δt is the control period, is the size of the i-th fleet, w thr is the waiting time threshold, c is the control Upper limit parameter.

[0082] The efficiency reward is designed to enable the strategy learned by the algorithm to generate a fleet size that improves traffic efficiency, expressed as the average fleet speed. With maximum speed v M The closer the speed is to the maximum speed, the higher the traffic efficiency is, and the greater the traffic efficiency reward fed back to the fleet is. The calculation is as follows:

[0083]

[0084] The safety reward is designed to enable the strategy learned by the algorithm to generate a convoy size that can reduce the risk of collision. It is expressed as the ratio of the distance between convoys to their corresponding safety distance. The smaller the spatial or temporal distance, the higher the risk of collision between convoys, and the smaller the safety reward should be. Safety Reward The calculation is as follows:

[0085]

[0086] Among them, S d (t) and S t (t) are the spatial and temporal distances between the two convoys, d p_thr and t p_thr are the spatial and temporal safety distance thresholds of the two convoys respectively.

[0087] In order to optimize the three goals of fairness, efficiency and safety at the same time, the total reward of the adaptive formation algorithm is calculated as follows:

[0088]

[0089] Among them, the corresponding weights of each adaptive formation target are: fairness weight ω pf , efficiency weight ω pe , safety weight ωps .

[0090] In the MADDPG algorithm, a fleet is considered as an intelligent agent, and each intelligent agent has an action network (actor network) and a value function network (critic network). Figure 1 As shown, the actor network transforms the local state observed by the agent in the environment ( or ) as input, calculates the corresponding action, and uses the action as the network output. The critic network will observe the global state ( or ) as input to estimate the Q value, and based on this value, the executed action is evaluated through the temporal difference error (TD-error), and at the same time fed back to the actor network to guide the actor network to optimize its strategy using gradient descent.

[0091] Both the actor network and the critic network consist of an input layer and three fully connected layers (such as Figure 2 The specific construction order of the actor network is:

[0092] (1) The input layer receives input data (the state information of n+1 agents) and performs layer normalization preprocessing.

[0093] (2) Fully connected layer (64 neurons), layer normalization, and ReLu activation function introduces nonlinear characteristics.

[0094] (3) Fully connected layer (64 neurons), layer normalization, and ReLu activation function introduces nonlinear characteristics.

[0095] (4) Fully connected layer (1 neuron), processed by Tanh activation function, to obtain the output action value.

[0096] The structure of the critic network differs in how it introduces the action space and processes the final output.

[0097] The specific construction order of the critic network is:

[0098] (1) The input layer receives input data (the state information of n+1 agents) and performs layer normalization preprocessing.

[0099] (2) Fully connected layer (64 neurons), layer normalization, and ReLu activation function introduces nonlinear characteristics.

[0100] (3) Introducing action space information (a t ) to enhance the network’s ability to assess action value.

[0101] (4) Fully connected layer (64 neurons), layer normalization, and ReLu activation function introduces nonlinear characteristics.

[0102] (5) Fully connected layer (1 neuron), directly obtain the output Q value.

[0103] S2. Use MADDPG to generate the optimal acceleration a of the leading vehicle in each target convoy according to the current unsignalized intersection environment, and use the following model IDM to generate the acceleration of the following vehicles in the convoy.

[0104] The fleet coordination MADDPG model also includes elements such as state, action, multi-objective reward function and network.

[0105] The state information of each agent is the information of the leading vehicle in the convoy, the position is the center position of the convoy, and the speed is the speed of the leading vehicle in the convoy. In the MADDPG algorithm that generates the acceleration of the target convoy leading vehicle, the input state is the state of the target fleet and its surrounding n decision reference fleets. The state of the target fleet at time t is expressed as The states of n decision reference fleets are expressed as follows:

[0106]

[0107] in, is the acceleration of the leading vehicle in the i-th convoy, is the acceleration of the leading vehicle in the nth decision reference convoy around convoy i.

[0108] The output of the coordinated action of the fleet is acceleration. In time step t, it is necessary to decide the acceleration of n+1 agents in the environment. The action space can be expressed as The action of agent i at time step t The calculation is as follows:

[0109]

[0110] Among them, π is the actor network control strategy, θ i represents the weight of the actor network, The status of the target fleet and the reference fleet for decision making.

[0111] MADDPG generates the acceleration of the lead vehicle in the convoy. The remaining vehicles in the convoy follow the lead vehicle. The acceleration is calculated as follows:

[0112]

[0113] in, is the expected following distance of vehicle j in fleet i; s 0is the minimum safety distance; v j is the current speed of the jth vehicle in the convoy i; T is the safe following distance; Δv j is the speed difference between the jth vehicle and the preceding vehicle in the convoy i; a max is the maximum acceleration; a c is the comfortable deceleration; δ is the sensitivity parameter of acceleration; is the following acceleration of the jth vehicle in the convoy i; v 0 is the initial velocity of the jth vehicle in the fleet i; Δs j is the distance between the jth vehicle and the vehicle in front.

[0114] Team coordination takes into account safety incentives s (t), traffic efficiency reward r e (t) and driving comfort reward r j (t).

[0115]

[0116] Among them, S d (t) and S t (t) are the spatial and temporal distances between the two convoys, (x i (t),y i (t)) represents the position of team i at time t, (x i-1 (t),y i-1 (t)) represents the position of team i-1 at time t, l i-1 is the length of the convoy i-1, and the convoy length is l = d last -d first It is calculated that d last Indicates the distance between the last vehicle in the convoy and the stop line, d first Indicates the distance between the first vehicle in the convoy and the stop line; (0,d p_thr ) represents the spatial distance range where there is a risk of collision, d p_thr =D thr +l i-1 , D thr is the safety distance threshold, v i (t) is the speed of team i at time t, t r is the delay time, including the driver's reaction time and the system's reaction time during braking, a max is the maximum braking acceleration; (0,t p_thr ) represents the time distance range where there is a risk of collision, v M is the maximum speed limit at the intersection, v m is the minimum speed limit at the intersection; at is the acceleration at the current moment, a t-1 is the acceleration at the previous moment, Δa max is the maximum value of the acceleration change.

[0117] If the team runs normally, the total reward It is the weighted sum of the rewards for each target. If the vehicle collides, the total reward is -10. The total reward is calculated as follows:

[0118]

[0119] The judgment conditions for convoy collision are as follows:

[0120]

[0121] Among them, l 1 and l 2 are the lengths of the two convoys respectively; d gap Represents the collision gap. If the collision condition is met, it is determined that the two convoys have collided.

[0122] Environmental interaction mechanisms for fleet coordination and network and Figure 2 The network is the same.

[0123] S3. Adjust the fleet according to N.

[0124] The actor network is based on the target fleet After deciding the optimal fleet size, you need to Make adjustments. If the size of the fleet is smaller than the size of the decision-making fleet, the fleet behind needs to be The vehicle information of and information; if If the size is larger than the fleet size of the decision, The excess vehicles are separated and reorganized into a fleet.

[0125] S4. Control the convoy’s traffic according to the acceleration generated by MADDPG and IDM.

[0126] The formula for controlling the vehicle to move forward is as follows:

[0127]

[0128] S5. Update the unsignalized intersection environment, update the status information, and the reward.

[0129] S6, Evaluation and Network Update: Each agent has an actor-critic network that generates the fleet size and an actor-critic network that generates acceleration. In the process of using the value function network to evaluate the value, not only the actions of the target agent are considered, but also the actions of the decision reference agent are integrated to comprehensively evaluate the coordination between the agents. The objective function of the MADDPG algorithm is as follows:

[0130]

[0131] Where: J(θ i ) represents the objective function of agent i; γ t is a discount factor used to weigh the importance of short-term and long-term rewards, with a value between 0 and 1; represents the reward obtained by agent i at time step t (if the network with the best size is generated, for If the network generates the optimal acceleration, the incoming Value is ); n represents the total number of decision reference agents; represents the policy function of each agent.

[0132] The present invention uses Python3.6 to build a two-way six-lane unsignaled intersection simulation scene, conducts a simulation experiment on the proposed unsignaled vehicle traffic control method, and compares the performance of different traffic control methods in terms of safety, traffic efficiency, driving comfort and training effect under the same simulation conditions.

[0133] (1) Simulation experiment setup.

[0134] The experiment of the present invention includes two stages: training and testing. The purpose of the training stage is to obtain the optimal control strategy. The following three methods are introduced in the testing stage for comparative analysis:

[0135] ① Uncontrolled vehicle passage (UVP) method: A 3s stop observation time is set when the vehicle approaches the stop line of the intersection. For vehicles traveling in the intersection control area, only the IDM model is used to follow the vehicle without using any guidance strategy.

[0136] ②CoMADDPG method: A single-vehicle control method based on cooperative MADDPG, which takes a single vehicle as the control object to control the passage of each vehicle.

[0137] ③ Fleet control based on fixed condition formation algorithm (PC-FCPA) method: The fixed condition formation algorithm is used to form a platoon. The size of the platoon remains unchanged during the driving process. This strategy uses a fixed platoon as the control object to control the platoon traffic.

[0138] (2) Experimental data.

[0139] The data used for training and testing in the present invention are all different traffic volume (number of vehicles / hour / lane) data generated in a simulation environment. The traffic volume data used for testing includes multiple different levels, and the data values ​​are from low to high: 200veh / hr / lane, 400veh / hr / lane, 600veh / hr / lane, 800veh / hr / lane, 1000veh / hr / lane, and 1200veh / hr / lane.

[0140] (3) Experimental results

[0141] The test results are shown in Table 1. It can be seen from Table 1 that the traffic control method proposed in the present invention can ensure a safe passing rate of more than 99%. Compared with UVP, CoMADDPG and PC-FCPA strategies, PC-APA strategy performs best in terms of traffic efficiency and driving comfort, with traffic efficiency increased by 56.35%, 10.26%, and 5.55% (the average of each traffic volume increase percentage), and driving comfort increased by 80.67%, 52.96%, and 23.93% (the average of each traffic volume increase percentage).

[0142] Table 1 Comparison of test results of various methods

[0143]

[0144]

[0145] Based on the same inventive concept, the present invention also proposes a vehicle traffic control system for an unsignalized intersection, comprising:

[0146] The acquisition module is used to obtain the current observation status of the target fleet and the decision reference fleet in the unsignalized intersection environment; the observation status includes the position, size, acceleration and lane position of the target fleet and the decision reference fleet; the decision reference fleet represents a fleet that has a conflicting relationship with the lane position of the target fleet.

[0147] The action calculation module is used to input the current observation states of the target fleet and the decision reference fleet into the actor network of the MADDPG algorithm to generate the action to be performed by the target fleet at the next moment; the action includes adjusting the size of the target fleet and the acceleration of the lead vehicle of the target fleet; according to the adjusted size of the target fleet and the acceleration of its lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet.

[0148] The optimization module is used to input the current observation status of the target fleet and the decision reference fleet, as well as the action to be performed by the target fleet at the next moment, into the critic network to obtain an estimated Q value; the action to be performed by the target fleet at the next moment is evaluated according to the Q value, and the evaluation result is fed back to the actor network and combined with the total reward, and the action to be performed by the target fleet at the next moment is continuously updated and optimized iteratively, and a strategy for adjusting the optimal fleet size and the optimal acceleration of the leading vehicle of the fleet is generated.

[0149] The control module is used to coordinate the control of the target fleet in the unsignalized intersection environment according to the strategy of adjusting the optimal fleet size and the optimal fleet head vehicle acceleration of the target fleet, as well as the acceleration of the vehicle following the head vehicle; wherein, within a control cycle, all fleets in the unsignalized intersection environment will successively become target fleets to execute actions, thereby completing the traffic control of all fleets in the unsignalized intersection.

[0150] Based on the same inventive concept, the present invention also proposes a computer device for controlling vehicle traffic at an unsignalized intersection, comprising: a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, the steps of the method for controlling vehicle traffic at an unsignalized intersection are implemented.

[0151] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for controlling vehicle traffic at an unsignalized intersection, characterized in that: The following steps are involved: Obtain the current observation status of the target fleet and decision reference fleet in the unsignalized intersection environment; The observed state includes the position, size, acceleration and lane position of the target fleet and the decision reference fleet; The decision reference fleet represents a fleet that has a conflicting lane position with the target fleet; The current observation states of the target fleet and the decision reference fleet are input into the actor network of the MADDPG algorithm to generate the action to be performed by the target fleet at the next moment; wherein the action includes adjusting the size of the target fleet and the acceleration of the lead vehicle of the target fleet; according to the adjusted size of the target fleet and the acceleration of its lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet; The current observation status of the target fleet and the decision reference fleet, as well as the action to be performed by the target fleet at the next moment, are input into the critic network to obtain an estimated Q value; the action to be performed by the target fleet at the next moment is evaluated based on the Q value, and the evaluation result is fed back to the actor network and combined with the total reward, the action to be performed by the target fleet at the next moment is continuously updated and optimized iteratively, and a strategy for adjusting the optimal fleet size and the optimal acceleration of the leading vehicle of the fleet is generated; The target fleet in the unsignalized intersection environment is coordinated and controlled according to the strategy of adjusting the optimal fleet size and the optimal fleet lead vehicle acceleration of the target fleet, as well as the acceleration of the vehicles following the lead vehicle; wherein, within one control cycle, all fleets in the unsignalized intersection environment will successively become the target fleet to execute actions, thereby completing the traffic control of all fleets in the unsignalized intersection.

2. A vehicle traffic control method at an unsignalized intersection according to claim 1, characterized in that: The acquisition of the current observation state of the target fleet and the decision reference fleet in the unsignalized intersection environment includes an observation state for generating the target fleet size and an observation state for generating the acceleration of the leading vehicle of the target fleet; the observation state for generating the target fleet size include: The state of the target fleet at time t is The decision reference fleet state is expressed as: in, is the position of team i, is the size of fleet i, is the speed of the leading vehicle in fleet i, is the lane number where team i is located, is the position of the nth decision reference fleet around fleet i, is the size of the nth decision reference fleet around fleet i, is the speed of the leading vehicle of the nth decision reference fleet around fleet i, The lane number of the nth decision reference fleet around fleet i; The observation state used to generate the acceleration of the lead vehicle in the target convoy include: The state of the target fleet at time t is expressed as: The decision reference fleet state is expressed as: in, is the acceleration of the leading vehicle in convoy i, is the acceleration of the leading vehicle in the nth decision reference convoy around convoy i.

3. The method for controlling vehicle traffic at an unsignalized intersection according to claim 1, characterized in that: The total reward includes a total reward for generating an optimal fleet size strategy with fairness, traffic efficiency and safety as optimization objectives, and a total reward for generating an optimal fleet lead vehicle acceleration strategy with safety, traffic efficiency and driving comfort as optimization objectives.

4. A vehicle traffic control method at an unsignalized intersection according to claim 3, characterized in that: The total reward for generating the optimal fleet size strategy with fairness, traffic efficiency and safety as optimization goals It is expressed as: Among them, ω pf Reward for fairness The weight, ω pe Indicates traffic efficiency reward The weight, ω ps Safety Reward The fairness reward is measured by the time each team stays in the control area. The longer the team stays in the control area, the smaller the fairness reward. It is expressed as: in: is the total time step that the first vehicle in the i-th fleet stays, is the size of the i-th fleet, w thr is the waiting time threshold, c is the control Upper limit parameters; represents the length of time the i-th convoy stays in the control area; The efficiency bonus is expressed as the average speed of the fleet With maximum speed v M The closer the speed is to the maximum speed, the higher the traffic efficiency is, and the greater the traffic efficiency reward fed back to the fleet is. It is expressed as: The safety reward is expressed as the ratio of the distance between the convoys to their corresponding safety distance. The smaller the spatial or temporal distance, the higher the risk of collision between the convoys, and the smaller the safety reward. It is expressed as: Among them, S d (t) and S t (t) are the spatial and temporal distances between the two convoys, d p_thr and t p_thr are the spatial and temporal safety distance thresholds of the two convoys respectively.

5. The method for controlling vehicle traffic at an unsignalized intersection according to claim 3, characterized in that: The total reward of the optimal acceleration strategy for the leading vehicle in the convoy is generated by taking safety, traffic efficiency and driving comfort as the optimization goals. It is expressed as: If the team runs normally, the total reward It is the weighted sum of the rewards for each target. If the team collides, the total reward is -10; The security reward r s (t), traffic efficiency reward r e (t) and driving comfort reward r j (t) are respectively expressed as: Among them, S d (t) and S t (t) are the spatial and temporal distances between the two convoys, (x i (t),y i (t)) represents the position of team i at time t, (x i-1 (t),y i-1 (t)) represents the position of team i-1 at time t, l i-1 is the length of the convoy i-1, and the convoy length is l = d last -d first It is calculated that d last Indicates the distance between the last vehicle in the convoy and the stop line, d first Indicates the distance between the first vehicle in the convoy and the stop line; (0,d p_thr ) represents the spatial distance range where there is a risk of collision, d p_thr =D thr +l i-1 , D thr is the safety distance threshold, v i (t) is the speed of team i at time t, t r is the delay time, including the driver's reaction time and the system's reaction time during braking, a max is the maximum braking acceleration; (0,t p_thr ) represents the time distance range where there is a risk of collision, v M is the maximum speed limit at the intersection, v m is the minimum speed limit at the intersection; a t is the acceleration at the current moment, a t-1 is the acceleration at the previous moment, Δa max is the maximum value of acceleration change; The judgment condition for the convoy to collide is: Among them, D p (t) is the distance between convoy p1 and convoy p2 at time t; (x p1 (t),y p1 (t)) and (x p2 (t),y p2 (t)) is the position of convoy p1 and convoy p2 at time t, l1 and l2 are the lengths of convoy p1 and convoy p2 respectively; d gap Represents the collision gap.

6. The method for controlling vehicle traffic at an unsignalized intersection according to claim 2, characterized in that: The action includes adjusting the size of the target convoy and the acceleration of the head vehicle of the target convoy; the action space of the size of the target convoy is expressed as: The actions of the team It is expressed as: Among them, θ l The actor network weights representing the size of the generated fleet, Indicates in status The action value calculated according to the strategy π; The action space of the acceleration of the target convoy head vehicle is expressed as The action of team i at time step t It is expressed as: Among them, π is the actor network control strategy; θ i Represents the weights of the actor network.

7. The method for controlling vehicle traffic at an unsignalized intersection according to claim 1, characterized in that: According to the adjusted target fleet size and the acceleration of the lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet. It is expressed as: in, is the expected following distance of vehicle j in platoon i; s0 is the minimum safe distance; v j is the current speed of the jth vehicle in the convoy i; T is the safe following distance; Δv j is the speed difference between the jth vehicle and the preceding vehicle in the convoy i; a max is the maximum acceleration; a c is the comfortable deceleration; δ is the sensitivity parameter of acceleration; is the following acceleration of the j-th vehicle in the convoy i; v0 is the initial velocity of the j-th vehicle in the convoy i; Δs j is the distance between the jth vehicle and the vehicle in front.

8. The method for controlling vehicle traffic at an unsignalized intersection according to claim 1, characterized in that: The method of coordinating and controlling the target convoy in an unsignalized intersection environment according to the strategy of adjusting the optimal convoy size and the optimal convoy head vehicle acceleration of the target convoy and the acceleration of the vehicle following the head vehicle specifically includes the following steps: If the target fleet If the size of the fleet is smaller than the optimal fleet size, the fleet behind The vehicle information of and information; If the target fleet If the size of the fleet is smaller than the optimal fleet size, the fleet behind The vehicle information is added to the target fleet and updated and information; like If the size is greater than the optimal fleet size, The excess vehicles are separated and reorganized into a fleet; The generated acceleration of the target vehicle and the acceleration of the following vehicle are used to control the passage of each vehicle in the convoy, which can be expressed as: in, represents the position of team i at time t, represents the position of team i at time t+Δt, Δt is the time of a control cycle, represents the acceleration of the ith team at time t, represents the speed of team i at time t, represents the speed of team i at time t+Δt.

9. A vehicle traffic control system for an unsignalized intersection, characterized in that: include: An acquisition module is used to obtain the current observation status of the target fleet and the decision reference fleet in an unsignalized intersection environment; The observed state includes the position, size, acceleration and lane position of the target fleet and the decision reference fleet; The decision reference fleet represents a fleet that has a conflicting lane position with the target fleet; The action calculation module is used to input the current observation states of the target fleet and the decision reference fleet into the actor network of the MADDPG algorithm to generate the action to be performed by the target fleet at the next moment; wherein the action includes adjusting the size of the target fleet and the acceleration of the lead vehicle of the target fleet; according to the adjusted size of the target fleet and the acceleration of its lead vehicle, the following model IDM is used to generate the acceleration of the vehicles following the lead vehicle in the target fleet; The optimization module is used to input the current observation status of the target fleet and the decision reference fleet, as well as the action to be performed by the target fleet at the next moment, into the critic network to obtain an estimated Q value; the action to be performed by the target fleet at the next moment is evaluated according to the Q value, and the evaluation result is fed back to the actor network and combined with the total reward, and the action to be performed by the target fleet at the next moment is continuously updated and optimized iteratively, and a strategy for adjusting the optimal fleet size and the optimal acceleration of the leading vehicle of the fleet is generated; The control module is used to coordinate the control of the target fleet in the unsignalized intersection environment according to the strategy of adjusting the optimal fleet size and the optimal fleet head vehicle acceleration of the target fleet, as well as the acceleration of the vehicle following the head vehicle; wherein, within a control cycle, all fleets in the unsignalized intersection environment will successively become target fleets to execute actions, thereby completing the traffic control of all fleets in the unsignalized intersection.

10. A computer device for controlling vehicle traffic at an unsignalized intersection, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the method for controlling vehicle traffic at an unsignalized intersection as described in any one of claims 1 to 8 when executing the computer program.

Citation Information

Patent Citations

  • Vehicle control method based on reinforcement learning control strategy in hybrid fleet

    CN112162555A

  • Single-point traffic signal control method based on intersection holographic data

    CN115691167A

  • Multi-agent cruise control method and device, electronic equipment and storage medium

    CN117048604A

  • Multi-vehicle formation decision-making method and system based on communication and multi-agent reinforcement learning

    CN117539254A

  • Non-signalized intersection vehicle passing sequence decision and trajectory optimization method and system

    CN118230583A

Cited By

  • Non-signalized intersection multi-agent scheduling optimization method based on deep reinforcement learning

    CN120108204A