Signal-free intersection multi-agent scheduling optimization method based on deep reinforcement learning

Abstract

The present application belongs to the field of intelligent transportation, and particularly relates to a multi-agent scheduling optimization method for a signal-free intersection based on deep reinforcement learning. The method comprises: building an online network and a target network based on an Actor-Critic structure for scheduling and control of intelligent networked vehicles; designing a joint reward function in combination with the characteristics of the signal-free intersection, and comprehensively considering factors such as a safety distance, a traffic efficiency, a speed deviation and a collision penalty; storing historical interaction data of the vehicles through an experience replay pool, breaking the time correlation and improving the sample utilization; through a deadlock detection and processing mechanism, ensuring that the vehicles avoid deadlock in a complex traffic environment; and using the trained model to provide an optimal scheduling strategy for each intelligent networked vehicle, including a driving speed and a passing time suggestion. The method effectively guarantees the traffic safety of the vehicles, reduces the waiting time and relieves traffic congestion, and realizes efficient scheduling and traffic control of the signal-free intersection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation, specifically relating to a multi-agent scheduling optimization method for unsignalized intersections based on deep reinforcement learning. Background Technology

[0002] With the increasing sophistication of intelligent connected transportation systems, urban road networks are facing growing challenges such as increased vehicle numbers and complex traffic flow. This is particularly evident at unsignalized intersections, where traditional timed control, sensor-based control, and adaptive control methods are insufficient to meet the demands of real-time, efficient traffic management. As autonomous driving and vehicle-to-infrastructure (V2I) technologies become more widespread, the efficient operation of intersections, as bottlenecks in traffic flow, directly impacts the smoothness and safety of the entire urban transportation system. However, unsignalized intersections, lacking traffic light guidance, are prone to vehicle weaving, traffic conflicts, and even deadlocks. Furthermore, they place higher demands on vehicle cooperation and scheduling, requiring a solution that can respond quickly while also optimizing the overall traffic flow.

[0003] Against this backdrop, multi-agent reinforcement learning technology is becoming a key breakthrough in traffic management at unsignalized intersections. By equipping each vehicle or group of vehicles with an independent agent, vehicles can collaboratively make decisions and plan driving strategies based on their own and surrounding environment information (such as position, speed, acceleration, and the state of neighboring vehicles), thereby achieving efficient scheduling and traffic optimization throughout the intersection. Among them, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm combines the advantages of deep learning and reinforcement learning, enabling each agent to learn a policy independently while maintaining a shared critic network for value evaluation, thus achieving stable and effective training in complex multi-agent interaction environments.

[0004] Compared to traditional signal control methods, the MADDPG-based multi-agent cooperative scheduling optimization method combines real-time performance with flexibility. On the one hand, vehicles can respond to changes in surrounding traffic information within seconds or even sub-seconds, accurately adjusting their speed or deciding on the order of passage. On the other hand, the agents' online learning mechanism can continuously optimize decision-making strategies based on dynamic changes such as traffic load and unexpected events, reducing the probability of vehicle delays and conflicts. Furthermore, by modeling and jointly optimizing interactions between vehicles, it can effectively avoid stagnation or congestion caused by vehicles competing for the same conflict zone, bringing a more efficient and safer traffic flow organization method to unsignalized intersections. Combined with vehicle-to-everything (V2X) technology and adaptive scheduling algorithms, information sharing and collaboration among agents are also smoother, which is of great significance for the development and upgrading of future urban transportation networks. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a multi-agent scheduling optimization method for unsignalized intersections based on deep reinforcement learning. By introducing a multi-agent reinforcement learning framework, distributed scheduling and speed control are performed on intelligent connected vehicles at unsignalized intersections. Under the premise of ensuring safety and avoiding deadlock, each agent learns driving strategies through a deep deterministic policy gradient method, achieving collaborative optimization of vehicle passage order and speed. This effectively reduces vehicle delay time at intersections, alleviates congestion, and improves overall traffic efficiency. This method fully utilizes sensing, communication, and positioning technologies to obtain real-time information about vehicles and the environment. Combined with a multi-agent collaborative decision-making mechanism, it can dynamically allocate right-of-way in complex traffic environments, reduce unnecessary waiting, and provide reliable speed control and scheduling strategies for intelligent connected transportation systems.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: a multi-agent scheduling optimization method for signalless intersections based on deep reinforcement learning, comprising the following steps:

[0007] S1. Initialize the agent: Initialize the motion state S(p,v,a,r) of each vehicle in the system, where p is the distance between the vehicle and the intersection conflict point; v is the current speed of the vehicle; a is the current acceleration of the vehicle; and r is the driving intention of the vehicle, including left turn, straight, and right turn. Through initialization, the system can accurately obtain the position information of the vehicle relative to the core area of ​​the intersection and its driving mode in the subsequent training and decision-making process, providing basic data for subsequent actions, including acceleration, deceleration, and decision-making.

[0008] S2. Construction of Online and Target Networks: In the multi-agent deep deterministic policy gradient (MADDPG) algorithm, each agent contains two main networks: an online network and a target network. The online network is responsible for predicting actions and estimating the value function in real time, while the target network is used to stabilize the training process and reduce learning oscillations.

[0009] Online network: This network consists of an Actor network and a Critic network; the current state S of the vehicle is used as the input of the Actor network, and after calculation by the Actor network, a deterministic action A is output, which is the predicted acceleration of the vehicle; the input of the Critic network consists of two parts, including state information S and action information A, and after network calculation, the value estimate Q(S,A) of taking action A in state S is output.

[0010] Target Network: This network consists of an Actor target network and a Critic target network; the Actor target network takes the next time step state S′ as input and outputs the next time step predicted action A′; (S′,A′) is used as the input of the Critic target network, and the Critic target network calculates and outputs the corresponding Q′(S′,A′);

[0011] The target network has the same structure as the online network, but its parameters are not synchronized with the online network in real time; instead, a soft update strategy is adopted, in which the parameters of the target network are exponentially smoothly transitioned to the parameters of the online network with only a small step size τ in each training.

[0012] S3. Update Critic Network Parameters: Update the parameters of the online Critic network based on the TD-error temporal difference error. Specifically, first, sample a batch of (s,a,r,s′,done) interaction samples from the empirical replay pool, then calculate the TD-error, and finally update the Critic network parameters by minimizing the TD-error. The TD-error formula is as follows:

[0013] TD-error=(R+γ·Q'(S',A')-Q(S,A))

[0014] Where Q′(S′,A′) is the next-time value estimate output by the target Critic network; Q(S,A) is the value prediction of the current action by the online Critic network; γ is the discount factor; and R is the immediate reward.

[0015] S4. Reward Function: The reward function includes the safety distance reward t. distance The penalty caused by changes in acceleration (jerk) and the incentive for improved traffic efficiency (d) distance Speed ​​deviation reward v reward And collision penalty, and by weighting or combining these elements, provide targeted feedback for each vehicle at unsignalized intersections;

[0016] S5. Experience Replay Pool: By storing, randomly sampling, and reusing the agent's interaction information in the environment, the stability of training and data utilization are improved. The specific steps are as follows:

[0017] S5-1. Initialize the buffer by creating a fixed-capacity storage structure using a double-ended queue to store the experience data generated in each interaction step.

[0018] S5-2. Store interaction information. After each step of environmental interaction, package the current vehicle state s, the action a, the reward r, the vehicle state s′ at the next moment, and whether it has ended as an experience tuple (s,a,r,s′,done) and append it to the buffer. When the buffer is full, remove the oldest experience to make room.

[0019] S5-3. Breaking data correlation, random sampling is used during the training phase to extract batch experience (s) from the replay pool. i ,a i,r i ,s i ′,done i Since random sampling is independent of the collection time sequence, it eliminates strong temporal correlation to a certain extent, making network training more stable.

[0020] S5-4. Multiple reuse and updates: In each round or every few training steps, the agent samples different batches of experience from the replay pool multiple times to update the policy network and value network.

[0021] S6. Deadlock Detection and Handling: To avoid vehicles forming a loop waiting phenomenon near the conflict point, a deadlock detection and handling mechanism is designed. After detecting that vehicles are forming a loop waiting, the deadlock state is promptly lifted by analyzing and adjusting the corresponding sequence or sign, so that the blocked vehicles can return to the normal passage sequence, ensuring the order and smooth flow of the overall traffic.

[0022] As a preferred embodiment of the present invention, the online network in S2 includes an Actor network and a Critic network, and the specific structures of the two networks are as follows:

[0023] S2.1, Actor Network Input Layer: The input layer has 20 nodes, covering the state information of the current vehicle and surrounding vehicles. First, layer normalization is performed to make the numerical scale of each feature consistent, and then the data is mapped to a 64-dimensional output.

[0024] S2.2, The first hidden layer of the Actor network: receives 64-dimensional features from the input layer, maps them through the first fully connected layer, and outputs 64-dimensional features;

[0025]

[0026] in n1 is the number of cells in this layer, and b1 is the bias;

[0027] S2.3, Second hidden layer of the Actor network: The 64-dimensional output of the first hidden layer is remapped to 1-dimensional features. The mapping process for h1 is as follows:

[0028] h2 = ReLU(W2h1 + b2)

[0029] in Further extract higher-order interaction features;

[0030] S2.4, Actor Network Output Layer: Maps the above 1D representation to an action scalar for predicting vehicle acceleration. The specific expression for the action scalar is as follows:

[0031] a raw =W3h2+b3

[0032] To ensure the output action is within the predetermined range, a hyperbolic tangent activation function is applied, and then multiplied by a constant w, where w = 3; thus obtaining the final action.

[0033] a=w·tanh(a raw )

[0034] Since the output range of tanh(·) is (-1,1), then action a∈(-3,3); this control ensures that the network output conforms to the normal traffic physical constraints;

[0035] S2.5, Critic Network Input Layer: Receives a 20-dimensional state vector S as input, and performs layer normalization to ensure consistency in numerical scale across different features; subsequently, the normalized state vector... Mapped to a hidden 64-dimensional dimension;

[0036] S2.6, First hidden layer of the Critic network: Let the above hidden state features be denoted as h. state Then, it is concatenated with the current action A, which has a dimension of 4, to form a fusion vector h. fusion h state with h fusion The specific formula is as follows:

[0037]

[0038] h fusion =concat(h state A)

[0039] Among them W state Let b be the weight matrix of the first hidden layer. state This is the bias term for the first hidden layer;

[0040] S2.7, Critic network second hidden layer: for h fusion By further applying fully connected mappings and activation functions, a deeper fusion representation h is obtained. fusion2 ;

[0041] h fusion2 =ReLU(W fusion h fusion +b fusion )

[0042] Among them W fusion Let b be the weight matrix of the second hidden layer. fusion This is the bias term for the second hidden layer;

[0043] S2.8, Critic network output layer: h is mapped through a linear layer. fusion2Mapped to a scalar Q value, it is used to evaluate the value under a given state and action, thereby providing a basis for action selection and policy updates;

[0044] Q(S,A)=W out h fusion2 +b out

[0045] Among them W out Let b be the output layer weight matrix. out This is the output layer bias term.

[0046] As a preferred embodiment of the present invention, the reward function in S4 is configured as follows, which provides effective feedback for multiple agents to learn the optimal strategy at unsignaled intersections:

[0047] S4.1, Safe Distance Bonus t distance :t distance It is the relative distance between the vehicle and the vehicle in front; if this distance is less than 4, the reward will increase, and the increase in reward depends on the reciprocal of this distance, which is calculated using the tanh function, indicating that the vehicle should try to keep as far away as possible to avoid a collision.

[0048]

[0049] The reward is negative; a smaller distance will result in a larger negative reward, encouraging vehicles to maintain a certain distance from the vehicle in front.

[0050] S4.2 Penalty for Acceleration Changes (jerk): Jerk is the rate of change of vehicle acceleration. Excessive acceleration changes can lead to unstable behavior, therefore a penalty is imposed.

[0051]

[0052] The severity of the punishment is proportional to the size of the jerk, with the aim of encouraging a smooth acceleration process. w =3;

[0053] S4.3, Traffic Efficiency Bonus d distance When the vehicle is at a spatial distance d from other potential conflict vehicles or target areas distance If the distance is too large, or if the vehicle successfully passes the conflict point, there is a positive incentive; if the distance is too small, a negative penalty is imposed to guide the vehicle to ensure the necessary safety margin.

[0054]

[0055] The reward is based on a logarithmic function of distance, encouraging vehicles to maintain a certain distance; vehicles that are too close or too far away will affect the reward.

[0056] S4.4, Speed ​​Deviation Reward vreward v is the current speed of the vehicle. m It is the minimum speed of the vehicle, a M and a m These are the maximum and minimum accelerations, respectively; this bonus is adjusted based on the vehicle's speed, with a greater bonus the closer the speed is to the target speed. w =2;

[0057]

[0058] S4.5 Collision Penalty: The penalty value when a collision occurs is -10. If no collision occurs, the reward continues to be calculated in the normal way.

[0059]

[0060] S4.6 Joint Reward R0: The rewards and penalties, such as safe distance reward, acceleration change penalty, traffic efficiency reward, speed deviation reward, and collision penalty, are summed up to evaluate and update the agent's overall performance at unsignalized intersections. To avoid extreme fluctuations in reward values ​​and ensure training stability, the total reward value for each iteration is limited to the range of [-20, 20].

[0061]

[0062] R = min(20, max(-20, R0)).

[0063] As a preferred embodiment of the present invention, the specific method for deadlock detection and handling in S6 is as follows:

[0064] S6.1 Deadlock Detection: During vehicle operation, when multiple vehicles are detected waiting for each other and none of them can proceed, forming a closed waiting loop, a deadlock is determined. The specific judgment conditions are as follows:

[0065] S6.1.1 Circular judgment of virtual preceding vehicle: During the vehicle's movement, each vehicle will record a "virtual preceding vehicle" identifier, which is the vehicle in front of the vehicle in the virtual lane sequence; when the system detects that a vehicle's "virtual preceding vehicle" is the same as its own, it indicates that the vehicle is circularly waiting in the queue.

[0066] S6.1.2 Detecting loops: Starting from a vehicle, search forward step by step along its "virtual preceding vehicle" chain. If you eventually return to the vehicle itself, it means that the current group of vehicles has formed a loop, i.e., a deadlock has occurred.

[0067] S6.1.3 Process all vehicles in the loop: Once a loop is detected, use a while flag: loop to set the lock flag of all vehicles in the loop to True to record that they are in a deadlock state; then, apply a forced unlocking strategy to the key vehicles to quickly break the mutual waiting situation and restore normal vehicle passage.

[0068] S6.2 Deadlock Handling: When a deadlock is detected, a specific mechanism is needed to unlock the vehicles in the loop to prevent them from getting stuck in a permanent waiting situation. The specific process is as follows:

[0069] S6.2.1, Deadlock detected and there is a lock a After the loop is confirmed, the lock flag of all vehicles in the loop is set to True; at the same time, at least one pair of vehicles is assigned the lock value. a =+1 and lock a =-1, used to distinguish vehicles that require manual adjustment during the unlocking phase;

[0070] S6.2.2 Deadlock Unlock Mechanism: In a deadlock state, the vehicle's actual acceleration target a According to lock a Make fine adjustments; where, if lock a =+1, the vehicle's acceleration is slightly increased from the current level; if locked a If the value is -1, a deceleration strategy is adopted to break the closed loop of mutual waiting;

[0071] target a =current a +lock a

[0072] S6.2.3 Reset Flag: After each simulation step, the system resets the vehicle's lock flag to False and sets the lock flag to False. a Reset to 0 to ensure that the status of each vehicle can be reassessed and the traffic policy updated in a timely manner in the next step.

[0073] The deep reinforcement learning-based multi-agent scheduling optimization method for signalless intersections described in this invention has the following technical advantages compared with existing technologies:

[0074] (1) This invention builds an Actor network and a Critic network. By constructing an Actor network (responsible for generating actions) and a Critic network (responsible for evaluating the value of actions) based on deep neural networks for each agent, it realizes refined decision-making for vehicles under different traffic conditions, thereby improving the accuracy of action prediction and training stability in unsignalized intersection scenarios.

[0075] (2) The present invention designs a combined reward system, which comprehensively sets a variety of reward and penalty factors for the vehicle's driving process at the intersection, including safe distance, acceleration change, speed deviation and collision penalty, so that the vehicle can maximize traffic efficiency while taking safety into account, and avoid overly conservative or aggressive driving strategies.

[0076] (3) The present invention designs a deadlock detection and deadlock handling mechanism. By detecting the loop of the virtual lane queue, it accurately identifies the deadlock state caused by vehicles waiting for each other and proposes a corresponding unlocking strategy: adjusting the acceleration of vehicles in the loop and resetting key markers, thereby quickly breaking the waiting loop, avoiding vehicles from being permanently blocked, and significantly improving the traffic efficiency of the intersection.

[0077] (4) This invention applies the MADDPG algorithm to realize multi-agent collaborative control. Each vehicle is regarded as an independent intelligent agent and is trained collaboratively. Through global information interaction and strategy sharing, vehicles in unsignalized intersections can be adaptively scheduled and yielded, taking into account the safety and efficiency of the overall traffic flow. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the process of the multi-agent scheduling optimization method for signalless intersections based on deep reinforcement learning, which is involved in this invention.

[0079] Figure 2 This invention relates to a virtual scenario of a two-way four-lane unsignaled intersection;

[0080] Figure 3 A graph showing the change in the distance between the vehicle and the point of conflict over time;

[0081] Figure 4 This is a graph showing the change of vehicle speed over time in each lane as described in the present invention. Detailed Implementation

[0082] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, some symbols in the embodiments of this application will be explained first, so that those skilled in the art can understand them.

[0083] like Figure 1 As shown, the multi-agent scheduling optimization method for signalless intersections based on deep reinforcement learning includes the following steps:

[0084] Step 1) (Initialize the agent): Create a virtual scene of a two-way four-lane unsignalized intersection, such as... Figure 2 As shown, the intelligent agent is initialized. The initial speed of each vehicle is set to 10 m / s, and its driving intentions include left turn, straight ahead, and right turn, mapped to 0, 1, and 2 respectively; the lane width is [lane]. cw =2.5m; Regarding the lane conflict settings, the lanes conflicting with lane 0 are [14,4,13,12,9,10,5], the lanes conflicting with lane 1 are [14,13,8,4,5,6,12], the lanes conflicting with lane 2 are [14,13,8,4,5,6,7], the lane conflicting with lane 3 is

[14] , the lanes conflicting with lane 4 are [2,8,1,0,13,14,9], the lanes conflicting with lane 5 are [2,1,12,8,9,10,0], the lanes conflicting with lane 6 are [2,1,12,8,9,10,11], and the lanes conflicting with lane 7 are

[14] . The lane that conflicts with lane 8 is [2], the lanes that conflict with lane 9 are [6,12,5,4,1,2,13], the lanes that conflict with lane 10 are [6,5,0,12,13,14,4], the lanes that conflict with lane 11 are [6], the lanes that conflict with lane 12 are [10,0,9,8,5,6,1], the lanes that conflict with lane 13 are [10,9,4,0,1,2,8], the lanes that conflict with lane 14 are [10,9,4,0,1,2,3], and the lanes that conflict with lane 15 are

[10] .

[0085] Step 2) (Online Network and Target Network): In the MADDPG algorithm, each agent contains two main networks: an online network and a target network. The online network is responsible for predicting actions and estimating the value function in real time, while the target network is used to stabilize the training process and reduce learning oscillations.

[0086] Step 2-1 (Training Process): Write the initial hyperparameters of the model into the args.txt file. This file contains (actor...) lr =0.0001, batch size =128, collision thr =2, critic lr =0.001, gamma=0.8, lane num =8. learn start =20000, mat path='arvTimeNewVeh_for_train.mat', model = 'MADDPG', num episodes =1, num units =64, o agentnum =4, save rate =1, seq maxstep =12, trans r =0.998) and other information. This includes key parameters such as the learning rate of the Actor and Critic networks, the source of training data, the number of iterations, and the collision threshold. Relevant data recorded during training is shown in Table 1.

[0087] Table 1 Training Data

[0088]

[0089] Step 2-2 (Testing Process): After training, the model parameters are saved in a file named "index.cptk". During testing, the model parameters are loaded from this file, and then the environment is evaluated and inference is performed. Relevant test results and key metrics are shown in Table 2.

[0090] Table 2 Test Data

[0091]

[0092]

[0093] Step 3) (Update Critic Network Parameters): Update the parameters of the online Critic network using TD-error calculation to make its estimation of state-action value (Q-value) more accurate. During the update process, the Critic network receives the current action and target value as input and adjusts its parameters according to the gradient descent optimization strategy. Specific process data is as follows: CurrentAction(act_batch):

[0094] [[0.27276698],[-0.46654604],[-0.13992999],[-0.03665939].[3. ],[0.14632147],[-0.11674444],[-0.01875685],[-0.01813661],[0.35182745],[-0.22684622],[3. ],[-0.0293106],[3.

[0095] ],[0.33290177],[-0.27257816],[3. ],[-0.02271797],[-0.06558337],[-0.37584846]]; TargetValue:

[0096] [[3.82066470e-01],[2.55410127e+00],[2.27651744e+00],[2.28416547e+00],[3.99474210e+00],[2 .37166658e+00],[2.28461303e+00],[2.28998521e+00],[2.45062300e+00],[-2.14768431e+01],[9.36 360753e-01],[3.99474273e+00],[2.12916730e+00],[-3.29848242e+01],[-6.52671537e+00],[5.6142 3004e-01],[1.82575739e+00],[1.27184735e-01],[-1.71530677e+01],[2.24428944e+00]]; TD-error:

[0097] [[0.52063529],[2.73378211],[2.45692952],[2.24299012],[3.98884936],[2.55387273],[2.26137023],[2.26192263],[2.59914314],[21.29501317], [0.95180851],[4.17391506],[2.31121454],[32.80309286],[6.34478727],[0.74383929],[1.78324885],[0.30675002],[16.97211773],[2.42467845]].

[0098] Step 4) (Reward Function): The combined reward R0 is calculated by integrating multiple reward and penalty factors, including safe distance reward, acceleration change penalty, traffic efficiency reward, speed deviation reward, and collision penalty. These rewards and penalties are designed to comprehensively consider the safety, efficiency, and comfort of vehicle traffic, and are summed to form the final combined reward. The specific update process is as follows:

[0099] Step 4-1)(Safe distance reward (t) distance )):t distance=[-2.163953413738653,-2.163953413738653,-2.163953413738653,-2.163953413738653,-2.163953413738653,-2.163953413738653,-2.163953413738653,-1.5355643918365445,-1.5354213840340718,-2.163953413738653,- 1.824152674000544,-1.8239931932137838,-2.163953413738653,-2.139516573207878,-2.1393429694247668,-2.163953413738653,-2.5451272764553807,-2.5449387688448177,-2.163953413738653,-2.9734626637786783];

[0100] Step 4-2)(Penalty for changes in acceleration):

[0101] jerk=[0.0005422948462178518,2.3761531073331455e-05,0.0010904396637960623,0.75,1.390420804396921 8e-05,0.75,-0.0,0.7689636595276929,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0,-0.0].

[0102] Step 4-3)(Traffic efficiency reward (d) distance )):d distance= [-1.11098113241536,-1.1258906855011732,-1.1408448250282024,-1.155843818380946,-1.170887935354866,-1.1859774481854297,-1.201112631577586,-1.2162937627356905,-1.2315211213938835,-1.2467949898469284,-1.2621156529815254,-1. The reward reflects the trend of the distance between the vehicle and the conflict point over time; a larger value indicates that the vehicle passed the intersection more efficiently. The specific continuous curve showing the distance between the vehicle and the conflict point over time is shown below. Figure 3 As shown.

[0103] Step 4-4)(Speed ​​Deviation Reward (v) reward )):v reward =[1.335395838283609,1.3400064516013799,1.3423126170078667,1.4333333333333336,1.3344579682992173,1.4333333333333336,1.533333333333339,1.326230702510608,1.5333333333333339,1.633333333333334,1.3384693101249834,1.63 The above results reflect the speed deviation reward of the vehicle at different time steps, and the continuous curves of the vehicle's speed over time in each lane are shown below. Figure 4As shown, the dynamic trend of vehicle speed is demonstrated, further verifying the effectiveness of speed deviation reward in optimizing vehicle traffic.

[0104] Steps 4-5) (Collision penalty): The penalty value is -10 when a collision occurs. If no collision occurs, the reward continues to be calculated in the normal way.

[0105] Steps 4-6) (Joint Reward R0): R0 = [-0.8296868037290686, -0.8285147851471206, -0.8291328877806594, -4.505515581336914, -5.3580333681808145, 0.01937991959468066, -5.214658417118433, -4.496415408530568, -0.630620080405319, -5.1744845] 723922905, -5.278033623826636, -0.5306200804053189, -5.135008453151371, -5.238557504585717, -0.43062008040531863, -5.096244799078844, -5.199793850513191, -0.3306200804053183, -5.058208680178575, -5.161757731612921).

[0106] Step 5) (Experience Replay Pool): By storing, randomly sampling, and reusing the agent's interaction information in the environment, the stability of training and data utilization are improved. The specific steps are as follows:

[0107] Step 5-1) Initialize the buffer: Maximum capacity of the buffer. size =500000; the empirical number of samples taken in each batch during sampling. size =32; Number of steps to collect before starting to learn start =2000; The total number of sampling steps desired during the entire training process is step=100000.

[0108] Step 5-2) (Storing Interaction Information): After each step of the environmental interaction, the current vehicle state s, the action a, the reward r, the vehicle state s′ at the next moment, and whether it has ended (done) are packaged into an experience tuple (s,a,r,s′,done) and appended to the buffer.

[0109] Step 5-3) (Breaking Data Correlation): During the training phase, random sampling is used to extract batch experience (s) from the replay pool. i ,a i ,r i ,s i ′,done i Since random sampling is independent of the collection time sequence, it can eliminate strong temporal correlation to a certain extent, making network training more stable.

[0110] Step 5-4) (Multiple Reuses and Updates): In each round or every few training steps, the agent can sample different batches of experience from the replay pool multiple times to update the policy network and the value network.

[0111] Step 6) (Deadlock Detection and Handling): When a deadlock is detected in the vehicle, the actual acceleration target of the vehicle is adjusted. a According to lock a Make fine adjustments, specifically adjusting the target data. a =[1.3725348111746987,2,2.372534811174699,2,3.372534811174699,2,4,2,4,2,4,2,-1.1066041013823538,1.2124865732709602,-2.106604101382354,2.21248657327096,-3.106604101382354,3.2124865 7327096,-4,4,-4,4,-4,4,-4,4,-4,4,-1.154050830921937,0.9427275818584404,-2.1540508309219373,1.9427275818584404,-3.1540508309219373,2.94272758185844,-4,3.94272758185844,-4,4,-4,4).

[0112] Validated through practical examples, this method can provide accurate driving speed and scheduling suggestions for each intelligent connected vehicle using the MADDPG algorithm. Through an intelligent multi-agent cooperative scheduling strategy, vehicles can achieve efficient and safe passage in unsignalized intersection environments, avoiding deadlocks and effectively reducing waiting time, thus improving overall traffic flow efficiency. This method significantly alleviates traffic congestion and provides an efficient and reliable solution for future intelligent transportation systems.

[0113] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-agent scheduling optimization method for signalless intersections based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Initialize the agent: Initialize the motion state S(p,v,a,r) of each vehicle in the system, where p is the distance between the vehicle and the intersection conflict point; v is the current speed of the vehicle; a is the current acceleration of the vehicle; and r is the driving intention of the vehicle, including left turn, straight, and right turn. Through initialization, the system can accurately obtain the position information of the vehicle relative to the core area of ​​the intersection and its driving mode in the subsequent training and decision-making process, providing basic data for subsequent actions, including acceleration, deceleration, and decision-making. S2. Construction of Online and Target Networks: In the multi-agent deep deterministic policy gradient (MADDPG) algorithm, each agent contains two main networks: an online network and a target network. The online network is responsible for predicting actions and estimating the value function in real time, while the target network is used to stabilize the training process and reduce learning oscillations. Online Network: This network consists of an Actor network and a Critic network. The vehicle's current state S serves as the input to the Actor network, which calculates and outputs a deterministic action A, representing the vehicle's predicted acceleration. The Critic network's input consists of two parts: state information S and action information A. After calculation, it outputs a value estimate Q(S,A) for taking action A in state S. The Actor and Critic network structures are as follows: S2.1, Actor Network Input Layer: The input layer has 20 nodes, covering the state information of the current vehicle and surrounding vehicles. First, layer normalization is performed to make the numerical scale of each feature consistent, and then the data is mapped to a 64-dimensional output. S2.2, The first hidden layer of the Actor network: receives 64-dimensional features from the input layer, maps them through the first fully connected layer, and outputs 64-dimensional features; ； in n1 is the number of cells in this layer, and b1 is the bias. ; S2.3, Second hidden layer of the Actor network: The 64-dimensional output of the first hidden layer is remapped to 1-dimensional features. The mapping process for h1 is as follows: ； in Further extract higher-order interaction features. , For bias; S2.4, Actor Network Output Layer: Maps the above 1D representation to an action scalar for predicting vehicle acceleration. The specific expression for the action scalar is as follows: ； Where W3 is the weight matrix of the Actor network output layer, and b3 is the bias term of the Actor network output layer; To ensure the output action is within a predetermined range, a hyperbolic tangent activation function is applied, and then multiplied by a constant w, where w=3; thus obtaining the final action. ； Since the output range of tanh(⋅) is (−1,1), then action a∈(−3,3); this control ensures that the network output conforms to the normal traffic physical constraints; S2.5, Critic Network Input Layer: Receives a 20-dimensional state vector S as input, and performs layer normalization to ensure consistency in numerical scale across different features; subsequently, the normalized state vector... Mapping to a hidden 64-dimensional dimension yields the hidden state features; S2.6, First hidden layer of the Critic network: The above hidden state features are denoted as... Then, it is concatenated with the current action A, which has a dimension of 4, to form a fusion vector. ; and The specific formula is as follows: ； ； in This is the weight matrix of the first hidden layer. This is the bias term for the first hidden layer; S2.7, Critic network second hidden layer: for By further applying fully connected mappings and activation functions, a deeper fusion representation is obtained. ; ； in This is the weight matrix of the second hidden layer. This is the bias term for the second hidden layer; S2.8, Critic network output layer: A linear mapping layer is used to... Mapped to a scalar Q value, it is used to evaluate the value under a given state and action, thereby providing a basis for action selection and policy updates; ； in This is the output layer weight matrix. For output layer bias terms; Target Network: This network consists of an Actor target network and a Critic target network; the input to the Actor target network is the state at the next time step. Output the predicted action for the next time step. ;Will As input to the Critic target network, the corresponding output is calculated through the Critic target network. ; The target network has the same structure as the online network, but its parameters are not synchronized with the online network in real time; instead, a soft update strategy is adopted, in which the parameters of the target network are exponentially smoothly transitioned to the parameters of the online network with only a small step size τ in each training. S3. Update Critic Network Parameters: Update the parameters of the online Critic network based on the TD-error temporal difference error. Specifically, first, sample a batch of (s,a,r,s′,done) interaction samples from the empirical replay pool, then calculate the TD-error, and finally update the Critic network parameters by minimizing the TD-error. The TD-error formula is as follows: ； in, Q(S,A) is the next-moment value estimate output by the target Critic network; Q(S,A) is the value prediction of the current action by the online Critic network. R represents the discount factor; R represents the instant reward. S4. Reward Function: The reward function includes the safety distance reward. Penalties arising from changes in acceleration (jerk) and incentives for improved traffic efficiency. Speed ​​deviation reward and collision penalties And by weighting or combining these factors, targeted feedback is provided for each vehicle at unsignalized intersections; the reward function is defined as follows: S4.1, Safe Distance Reward : It is the relative distance between the vehicle and the vehicle in front; if this distance is less than 4, the reward will increase, and the increase in reward depends on the reciprocal of this distance, which is calculated using the tanh function, indicating that the vehicle should try to keep as far away as possible to avoid a collision; ； The reward is negative; a smaller distance will result in a larger negative reward, encouraging vehicles to maintain a certain distance from the vehicle in front. S4.2 Penalty for Acceleration Changes: jerk is the rate of change of vehicle acceleration. Excessive acceleration changes can lead to unstable behavior, therefore a penalty is imposed. ； The severity of the punishment is proportional to the size of the jerk, with the aim of encouraging a smooth acceleration process. =3; Δt is the time interval between two adjacent simulation steps; S4.3, Traffic Efficiency Bonus When the vehicle is spatially distant from other potential conflict vehicles or target areas If the distance is too large, or if the vehicle successfully passes the conflict point, there is a positive incentive; if the distance is too small, a negative penalty is imposed to guide the vehicle to ensure the necessary safety margin. ； The reward is based on a logarithmic function of distance, encouraging vehicles to maintain a certain distance; vehicles that are too close or too far away will affect the reward. S4.4, Speed ​​Deviation Reward v is the current speed of the vehicle. It is the vehicle's minimum speed. and These are the maximum and minimum accelerations, respectively; this bonus is adjusted based on the vehicle's speed, with a greater bonus the closer the speed is to the target speed. =2; ； S4.5 Collision Penalty The penalty for a collision is -10; if no collision occurs, the reward is calculated normally. ； S4.6, Joint Rewards The rewards and penalties for safe distance, acceleration change, traffic efficiency, speed deviation, and collision are summed to evaluate and update the agent's overall performance at unsignaled intersections. To avoid extreme fluctuations in reward values ​​and ensure training stability, the total reward value for each iteration is limited to the range [−20, 20]. ； ； S5. Experience Replay Pool: By storing, randomly sampling, and reusing the agent's interaction information in the environment, the stability of training and data utilization are improved. The specific steps are as follows: S5-1. Initialize the buffer by creating a fixed-capacity storage structure using a double-ended queue to store the experience data generated in each interaction step. S5-2. Store interaction information. After each step of environmental interaction, pack the current vehicle state s, the action a, the reward r, the vehicle state s′ at the next moment, and whether it has ended into a done tuple (s,a,r,s′,done) and append it to the buffer. When the buffer is full, remove the oldest experience to make room. S5-3. Breaking data correlation, random sampling is used during the training phase to extract batch experience (s) from the replay pool. i ,a i ,r i ,s i ′,done i Since random sampling is independent of the collection time sequence, it eliminates strong temporal correlation to a certain extent, making network training more stable. S5-4. Multiple reuse and updates: In each round or every few training steps, the agent samples different batches of experience from the replay pool multiple times to update the policy network and value network. S6. Deadlock Detection and Handling: To avoid vehicles forming a loop waiting phenomenon near the conflict point, a deadlock detection and handling mechanism is designed. After detecting that vehicles are forming a loop waiting, the deadlock state is promptly lifted by analyzing and adjusting the corresponding sequence or sign, so that the blocked vehicles can return to the normal passage sequence, ensuring the order and smooth flow of the overall traffic.

2. The method for multi-agent scheduling optimization at signalless intersections based on deep reinforcement learning according to claim 1, characterized in that, The specific methods for deadlock detection and handling in S6 are as follows: S6.1 Deadlock Detection: During vehicle operation, when multiple vehicles are detected waiting for each other and none of them can proceed, forming a closed waiting loop, a deadlock is determined. The specific judgment conditions are as follows: S6.1.1 Circular judgment of virtual preceding vehicle: During the vehicle's movement, each vehicle will record a "virtual preceding vehicle" identifier, which is the vehicle in front of the vehicle in the virtual lane sequence; when the system detects that a vehicle's "virtual preceding vehicle" is the same as its own, it indicates that the vehicle is circularly waiting in the queue. S6.1.2 Detecting loops: Starting from a vehicle, search forward step by step along its "virtual preceding vehicle" chain. If you eventually return to the vehicle itself, it means that the current group of vehicles has formed a loop, i.e., a deadlock has occurred. S6.1.3 Process all vehicles in the loop: Once a loop is detected, use a while flag: loop to set the lock flag of all vehicles in the loop to True to record that they are in a deadlock state; then, apply a forced unlocking strategy to the key vehicles to quickly break the mutual waiting situation and restore normal vehicle passage. S6.2 Deadlock Handling: When a deadlock is detected, a specific mechanism is needed to unlock the vehicles in the loop to prevent them from getting stuck in a permanent waiting situation. The specific process is as follows: S6.2.1, Deadlock detected and there is Once the ring road is confirmed, all vehicles within the ring road... The flag is uniformly set to True; at the same time, at least one pair of vehicles is assigned a value. and This is used to distinguish vehicles that require manual adjustment during the unlocking phase; S6.2.2 Deadlock Unlock Mechanism: In a deadlock state, the vehicle's actual acceleration... in accordance with Make fine adjustments; among them, if The vehicle makes a slight upward adjustment based on its current acceleration; if Then, a deceleration strategy is adopted to break the closed loop of mutual waiting; ； Where, current a This is the vehicle's current acceleration; S6.2.3 Reset Mark: After each simulation step, the system will reset the vehicle's marker. Flag reset and will Reset to 0 to ensure that the status of each vehicle can be reassessed and the traffic policy updated in a timely manner in the next step.

Images