Crossing queue quick evacuation control method based on meta-reinforcement learning

Abstract

The application discloses a kind of based on meta-reinforcement learning intersection queuing quick evacuation control method, comprising: obtaining the traffic data of each entrance of intersection;Calculate the queuing rate of each entrance, judge whether there is sudden congestion, exist then execute next step, do not exist then execute normal scheme;Build intersection queuing quick evacuation meta-reinforcement learning model;Design the input, output and reward function of intersection queuing quick evacuation meta-reinforcement learning model;Meta-training is carried out to intersection queuing quick evacuation meta-reinforcement learning model, and the optimal meta initial parameter is obtained;Execute traffic signal control decision, generate next execution phase.The application predicts the evolution trend of queuing, intelligently switches signal control decision cloud reinforcement learning model under normal and congestion state, generates phase sequence with the fastest global dissipation time as target, to realize the quick evacuation of intersection queuing vehicle, with the quick identification and adaptive capacity to sudden congestion.

Description

Technical Field

[0001] This invention relates to the field of urban intelligent traffic signal control technology, and in particular to a method for rapid queuing and evacuation control at intersections based on meta-reinforcement learning. Background Technology

[0002] In urban transportation networks, intersections, as the meeting points of multiple roads, undertake complex traffic functions such as vehicle turning, divergence, and merging, and are key bottlenecks in the entire road network. Studies show that approximately 30%-40% of delays in urban traffic occur at and around intersections, making intersection signal control one of the core technical means to improve the overall efficiency of the road network and alleviate traffic congestion.

[0003] Traditional intersection signal control methods, such as timing control and sensor control, while meeting basic traffic management needs to some extent, are inherently limited in their ability to effectively address the highly dynamic, random, and complex nature of modern urban traffic flow. To overcome these limitations, researchers have introduced advanced technologies such as artificial intelligence, machine learning, and reinforcement learning into the field of traffic signal control, driving the development of intelligent signal control technology. Reinforcement learning, through continuous interaction between the agent and its environment, learns optimal control strategies via a trial-and-error mechanism, and is widely used in signal control due to its powerful sequential decision-making capabilities. Algorithms such as Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG) have shown great potential in this field. These methods can learn complex mapping relationships from high-dimensional traffic state data and output better signal control commands. Meta-learning is a cutting-edge branch of machine learning, aiming to enable models to quickly adapt to new tasks. When encountering unseen traffic patterns, the agent can utilize its internalized "learning ability" to quickly adjust its strategy with only a few samples or update steps, demonstrating excellent adaptability and robustness. In recent years, some studies have attempted to apply meta-reinforcement learning to intersection decision-making for intelligent vehicles, enabling them to quickly adapt to the uncertainty of the motion state of surrounding vehicles.

[0004] Nevertheless, existing single-intersection signal control methods, including traditional and emerging intelligent methods, all have limitations to varying degrees when dealing with sudden, large-scale queuing scenarios: 1) Traditional methods rely on fixed rules or simple reactions, lacking flexibility and foresight. 2) While standard reinforcement learning methods have strong learning capabilities, they are costly to train, and the learned strategies are often strongly correlated with the training data, exhibiting poor generalization ability when facing sudden out-of-distribution congestion and failing to achieve rapid adaptation. 3) Existing meta-learning applications mostly focus on vehicle-level decision-making rather than traffic light-level optimization control, and have not specifically optimized algorithms and architectures for the goal of rapid queuing evacuation.

[0005] Therefore, there is an urgent need for a novel intelligent signal control method specifically designed for rapid evacuation, capable of handling various unknown sudden congestion scenarios, and possessing strong generalization and rapid adaptive capabilities. This invention is proposed against this backdrop, aiming to enable signal control strategies to inherently possess a keen perception and rapid response capability to sudden congestion patterns through an innovative meta-learning framework and a highly targeted reward function design. This significantly improves the queuing and evacuation efficiency of single intersections under extreme conditions, filling a gap in existing technologies. Summary of the Invention

[0006] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a fast evacuation control method for intersection queues based on meta-reinforcement learning.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A fast evacuation control method for intersection queues based on meta-reinforcement learning includes the following steps:

[0009] S1 acquires traffic data for each approach lane at the intersection;

[0010] S2 calculates the queuing change rate of each entrance lane to determine whether there is a sudden congestion. If so, S3 is executed; otherwise, the normal plan is executed.

[0011] S3 constructs a meta-reinforcement learning model for rapid queuing and evacuation at intersections;

[0012] S4 designs the input, output, and reward functions of a reinforcement learning model for rapid queuing and evacuation at intersections;

[0013] S5 performs meta-training on the meta-reinforcement learning model for rapid queuing and evacuation at intersections to obtain the optimal meta-initial parameters.

[0014] S6 executes traffic signal control decisions and generates the next execution phase.

[0015] Furthermore, the traffic data in S1 includes intersection channelization information, the number of approach lanes, the queue length of each approach lane, and the turning flow of straight, left-turning, and right-turning traffic on each approach lane.

[0016] Furthermore, S2 calculates the queuing change rate of each entrance lane to determine whether there is a sudden congestion, specifically as follows:

[0017] The rate of change of the queue at current time t for import lane i i for:

[0018]

[0019] In the formula, Q t,i Q represents the number of vehicles queuing at time t in lane i;t-1,i Let be the number of vehicles queuing at time t-1 in the entrance lane; Δt is the time step, i.e., the time difference between the current time t and the next time t+1.

[0020] A sudden congestion occurs when the queuing change rate of any entrance lane exceeds a threshold of 5 for two consecutive time steps; otherwise, it does not.

[0021] Furthermore, the construction of the intersection queuing rapid evacuation meta-reinforcement learning model described in S3 includes defining multiple different traffic congestion tasks p(T), each task representing an intersection congestion scenario, each intersection congestion scenario including multiple sets of data, each set of data including three types of data: model input, model output, and reward function.

[0022] Furthermore, the intersection queuing rapid evacuation meta-reinforcement learning model is specifically a deep neural network containing a signal control strategy π(θ), where θ is the network's meta-initial parameter.

[0023] Furthermore, based on the current number of vehicles in the queue, the predicted number of vehicles in the queue, the pressure gradient, and the current phase, an enhanced traffic state representation E is constructed from the model input. t Specifically:

[0024] Construct the current queuing matrix Q for the entrance lane t :

[0025] Q t =(Q t,1 Q t,2 ,...,Q t,i ,...,Q t,n )

[0026] In the formula, n is the number of approach lanes at the intersection; Q t,n For vehicles queuing at current time t in lane n;

[0027] Constructing the entrance lane prediction queuing matrix Q t+1 :

[0028] For any entrance lane i, the predicted number of vehicles queuing at the next time step t+1 is Q. t+1,i for:

[0029] Q t+1,i =Q t,i +A t,i -D t,i

[0030] A t,i =q t,i ·Δt·f ij

[0031] D t,i =S i ·Δt·δi,green

[0032] In the formula, Q t+1,i Let A be the predicted number of vehicles queuing at the next time step t+1 for lane i; t,i D represents the estimated number of vehicles arriving at the current time t in lane i; t,i q represents the estimated number of vehicles leaving the import lane at the current time t; t,i f is the average flow rate of inlet channel i at current time t; ij For the steering ratio system, S represents the proportion of vehicles moving from entrance lane i to the target exit lane j; i δ is the saturation flow rate of inlet channel i; i,green It is a value of 0 or 1. When the candidate phase of the next cycle gives the green light to the inlet lane i, δ i,green =1, otherwise, δ i,green =0;

[0033] Then the entrance lane prediction queuing matrix

[0034] Constructing the pressure gradient matrix

[0035]

[0036] In the formula, Let be the pressure gradient of the inlet channel i at the current time t.

[0037]

[0038] In the formula, Q t-1 Let i be the number of vehicles queuing at the previous time t-1 in the import lane i.

[0039] Construct the current phase encoding matrix Ph t :

[0040] Ph t =(Ph1,Ph2,...,Ph m )

[0041] In the formula, m is the number of phases, Ph m This is the m-th phase;

[0042] Traffic conditions are integrated and spliced ​​to construct an enhanced traffic condition representation E t :

[0043]

[0044] Furthermore, a signal phase set is constructed using a phase selection mode as the output action, and the signal phase set is a. t ={a1,a2,...,a m};

[0045] A reward function based on the global dissipation time is established, as shown in the following formula:

[0046]

[0047] In the formula, R t The reward function for the rapid evacuation signal control decision model is the sum of the weights of the total predicted number of queuing vehicles at all entrances at the next time t+1 and the absolute value of the total pressure gradient of all entrances at the current time t; α and β are weight coefficients.

[0048] Furthermore, the meta-reinforcement learning model for rapid evacuation of queuing at intersections is meta-trained to obtain the optimal meta-initial parameters. Specifically, a meta-learning algorithm is used for two-stage training: inner loop adaptation and outer loop meta-optimization.

[0049] Furthermore, the specific training process is as follows:

[0050] Initialize the meta-parameter θ;

[0051] A batch of tasks {T1, T2, ... T} is sampled from multiple different traffic congestion tasks p(T). u ,...,T N}, where N is the number of tasks; for each task T u The data is divided into a support set and a query set;

[0052] For each task T u The inner loop adaptation is performed: starting with the current initial parameters θ, the reward function and policy gradient are calculated using the support set data, and gradient descent is performed on the policy gradient to obtain the adaptation parameters θ for this task. u ;

[0053] For all tasks, perform outer loop optimization: based on the adaptation parameter θ for each task. u The loss value L for each task is calculated using the query set data. u The total loss function L is obtained by summing the loss values ​​of all tasks. loss The primary optimization objective is to minimize the total loss function, and the primary initial parameters θ are updated through gradient descent.

[0054] Until the initial parameters converge to θ, the optimal initial parameters θ obtained after training are obtained. * and signal control strategy π(θ) * ).

[0055] Furthermore, S6 executes traffic signal control decisions and generates the next execution phase, specifically as follows:

[0056] S6.1 The trained optimal initial parameters θ *Load the rapid evacuation model for queuing at intersections into a meta-reinforcement learning model;

[0057] S6.2 Collect traffic data at the intersection as a support set for the new task, and apply this data to the model parameters θ. * Perform gradient updates to generate the adapted optimal parameters θ. ** ;

[0058] S6.3 uses an adapted intersection queuing fast evacuation meta-reinforcement learning model to perform real-time signal control at the intersection. Based on the current model input and reward function, it generates the optimal signal control strategy π(θ). ** The optimal phase action is output and used as the next phase.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] 1. The traffic state matrix constructed in this invention innovatively integrates current queuing information, predicted queuing information, and pressure gradient, providing a data foundation for making more forward-looking traffic management decisions, thereby improving the efficiency of traffic signal control.

[0061] 2. This invention establishes a threshold triggering mechanism based on the dynamic index of queuing change rate, which can keenly capture changes in the situation at the early stage of congestion and quickly activate the rapid evacuation signal control decision model, realizing a millisecond-level switch from normal to congested state, thereby avoiding the problem of missing the best evacuation opportunity due to decision lag.

[0062] 3. This invention constructs a signal control decision model based on meta-learning, and innovatively designs a reward function with the direct objective of minimizing global dissipation time. It has advanced meta-ability to learn how to quickly evacuate. When it encounters a new sudden congestion pattern that was not seen during training, it can use its internalized learning ability to adapt rapidly, thereby achieving excellent generalization and robustness. Attached Figure Description

[0063] Figure 1 The flowchart of the intersection queuing rapid evacuation control method based on meta-reinforcement learning provided in the embodiments of the present invention is shown.

[0064] Figure 2 This is an intersection channelization information diagram provided for an embodiment of the present invention.

[0065] Figure 3 The diagram shows the turning flow rate of each approach lane at an intersection, as provided in an embodiment of the present invention.

[0066] Figure 4 A flowchart for constructing traffic state representations provided in embodiments of the present invention.

[0067] Figure 5The flowchart of the meta-training process of the cross-intersection queuing rapid evacuation meta-reinforcement learning model provided in the embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0069] Please see Figures 1-5 As shown, the present invention provides a fast evacuation control method for intersection queues based on meta-reinforcement learning, comprising the following steps:

[0070] S1. Obtain traffic data for each approach lane at the intersection.

[0071] The traffic data obtained for each approach lane of the intersection includes intersection channelization information, number of approach lanes, queue length of each approach lane, and turning flow for straight, left, and right turns at each approach lane.

[0072] In some embodiments of the present invention, intersection channelization information such as Figure 2 As shown, the intersection has a total of 12 approach lanes. The east approach has 3 through lanes and 1 left-turn lane; the south approach has 1 through lane and 1 left-turn lane; the west approach has 3 through lanes and 1 left-turn lane; and the north approach has 1 through lane and 1 left-turn lane. The turning flow rates for through, left, and right turns at each approach lane are as follows: Figure 3 As shown.

[0073] S2. Calculate the queuing change rate of each entrance lane to determine if there is a sudden congestion. If so, proceed to S3; otherwise, proceed to the normal plan.

[0074] The rate of change of the queue at current time t for import lane i i for:

[0075]

[0076] In the formula, Q t,i Q represents the number of vehicles queuing at time t in lane i; t-1,i Let Δt be the number of vehicles queuing at time t-1 in the entrance lane; Δt is the time step, i.e., the time difference between the current time t and the next time t+1.

[0077] A sudden congestion occurs when the queuing change rate of any entrance lane exceeds a threshold of 5 for two consecutive time steps; otherwise, it does not.

[0078] When there is a sudden congestion, proceed to step S3; when there is no sudden congestion, proceed with the normal procedure.

[0079] S3. Construct a meta-reinforcement learning model for rapid queuing and evacuation at intersections.

[0080] The construction of the intersection queuing fast evacuation meta-reinforcement learning model includes defining multiple different traffic congestion tasks p(T), each task representing an intersection congestion scenario in a specific time period, each scenario including multiple sets of metadata, each set of metadata including three types of data: input, output, and reward function; the intersection queuing fast evacuation meta-reinforcement learning model is a deep neural network containing a signal control policy π(θ), where θ is the meta-initial parameter of the network.

[0081] In some embodiments of the present invention, hundreds of intersection tasks under different congestion states are generated using SUMO simulation software. These tasks vary in the number of lanes, vehicle arrival rate, and vehicle flow ratio, and simulate different scenario modes such as morning rush hour, evening rush hour, and abnormal events. The signal control strategy π(θ) employs a two-layer GRU network containing 128 neurons to reflect the temporal dependence of traffic flow.

[0082] S4. Design the input, output, and reward functions of the model.

[0083] The model takes as input an enhanced traffic state representation and outputs as a set of signal phases constructed using a phase selection mode.

[0084] In some embodiments of the present invention, the input enhanced traffic state is represented as a 36-dimensional vector, including the current queue length of the 12 approach lanes (12-dimensional), the predicted queue length of the 12 approach lanes (12-dimensional), the pressure gradient of the 12 approach lanes (12-dimensional), and the current phase (4-dimensional, one-hot encoded). The output is a set of 4 phases {east-west straight, east-west left turn, north-south straight, north-south left turn}, with the green light time for each phase set according to the actual traffic flow. The weighting coefficients of the reward function are α = 0.6 and β = 0.4.

[0085] S5. Perform meta-training on the model to obtain the optimal meta-initial parameters.

[0086] The meta-training of the model involves a two-stage training process using a meta-learning algorithm, consisting of inner loop adaptation and outer loop meta-optimization, to obtain the optimal meta-initial parameters θ. * This includes the following steps:

[0087] S5.1 Initialize the meta-parameter θ, setting the inner loop learning rate to 0.01 and the outer loop learning rate to 0.001.

[0088] S5.2 Sample 8 tasks (N=8) from multiple different traffic congestion tasks p(T); for each task T u It samples two mutually exclusive sets of data from its experience replay cache, dividing them into a support set and a query set.

[0089] S5.3 for each task T uThe inner loop adaptation is performed: starting with the current initial parameters θ, the reward function and policy gradient are calculated using the support set data, and gradient descent is performed on the policy gradient to obtain the adaptation parameters θ for this task. u .

[0090] S5.4 performs outer loop optimization for all tasks: based on the adaptation parameter θ for each task. u The loss value L for each task is calculated using the query set data. u The total loss function L is obtained by summing the loss values ​​of all tasks. loss The primary optimization objective is to minimize the total loss function, and the primary initial parameters θ are updated through gradient descent.

[0091] S5.5 Repeat steps S5.3 to S5.4 until the initial meta-parameters converge to θ, thus obtaining the optimal initial meta-parameters θ after training. * and signal control strategy π(θ) * ).

[0092] S6. Execute traffic signal control decisions and generate the next execution phase.

[0093] The execution of traffic signal control decisions and the generation of the next execution phase include the following steps:

[0094] S6.1 The trained optimal initial parameters θ * Load the rapid evacuation model for queuing at intersections into a meta-reinforcement learning model;

[0095] S6.2 collects recent traffic data from the intersection as a support set for the new task, and applies the same algorithm as in S5.3 to the model parameters θ. * Perform a small number of gradient updates to generate the adapted optimal parameters θ. ** ;

[0096] S6.3 uses an adapted intersection queuing fast evacuation meta-reinforcement learning model to perform real-time signal control at the intersection, based on the current state E. t The optimal phase action is output by the reward function, and the next execution phase is generated.

[0097] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for fast evacuation control of intersection queues based on meta-reinforcement learning, characterized in that, Includes the following steps: S1 acquires traffic data for each approach lane at the intersection; S2 calculates the queuing change rate of each entrance lane to determine whether there is a sudden congestion. If so, S3 is executed; otherwise, the normal plan is executed. S3 constructs a meta-reinforcement learning model for rapid queuing and evacuation at intersections; S4 designs the input, output, and reward functions of a reinforcement learning model for rapid queuing and evacuation at intersections; S5 performs meta-training on the meta-reinforcement learning model for rapid queuing and evacuation at intersections to obtain the optimal meta-initial parameters. S6 executes traffic signal control decisions and generates the next execution phase; Based on the current number of vehicles in the queue, the predicted number of vehicles in the queue, the pressure gradient, and the current phase, an enhanced traffic state representation is constructed from the model input. Specifically: Construct the current queuing matrix for the entrance lane : In the formula, n is the number of approach lanes at the intersection; For vehicles queuing at current time t in lane n; Constructing an entrance lane prediction queuing matrix : For any entrance lane i, the predicted number of vehicles queuing at the next time step t+1. for: In the formula, Let i be the predicted number of vehicles queuing at the next time step t+1 in the import lane. Let i be the number of vehicles expected to arrive at the current time t in the import lane; The estimated number of vehicles leaving the import lane at the current time t; The average flow rate of inlet channel i at the current time t; For the steering ratio system, it represents the proportion of vehicles moving from entrance lane i to target exit lane j; Let i be the saturation flow rate of the inlet channel i; It is a value of 0 or 1, when the candidate phase of the next cycle gives the green light to the inlet lane i. ,otherwise, ; Then the entrance lane prediction queuing matrix , ; Constructing the pressure gradient matrix : In the formula, Let i be the pressure gradient at the current time t in the inlet channel; In the formula, Let i be the number of vehicles queuing at the previous time t-1 in the import lane. For time step; Construct the current phase encoding matrix : In the formula, m represents the number of phases. This is the m-th phase; Traffic conditions are integrated and spliced ​​together to construct an enhanced traffic condition representation. : 。 2. The method for rapid evacuation control of intersection queues according to claim 1, characterized in that, The traffic data in S1 includes intersection channelization information, the number of approach lanes, the queue length of each approach lane, and the turning flow of straight, left-turning, and right-turning traffic on each approach lane.

3. The method for rapid evacuation control of intersection queues according to claim 1, characterized in that, S2 calculates the queuing change rate of each entrance lane to determine whether there is a sudden congestion, specifically as follows: Rate of change of queue length of import lane i at current time t for: In the formula, Let i be the number of vehicles queuing at current time t in the import lane i. Let i be the number of vehicles queuing at time t-1 in the import lane; The time step is the time difference between the current time t and the next time t+1. When the queuing change rate of any entrance exceeds the threshold for two consecutive time steps. There may be sudden congestion at certain times; otherwise, there may be no congestion.

4. The method for rapid evacuation control of intersection queues according to claim 1, characterized in that, S3 describes the construction of a meta-reinforcement learning model for rapid queuing dispersal at intersections, including defining multiple different traffic congestion tasks. Each task represents a traffic congestion scenario at an intersection. Each traffic congestion scenario includes multiple sets of data, and each set of data includes three types of data: model input, model output, and reward function.

5. The method for rapid evacuation control of intersection queues according to claim 4, characterized in that, The aforementioned intersection queuing rapid evacuation meta-reinforcement learning model is specifically a model that includes signal control strategies. Deep neural networks, in which These are the initial parameters of the network.

6. The method for rapid evacuation control of intersection queues according to claim 1, characterized in that, A signal phase set is constructed using phase selection mode as the output action, and the signal phase set is: ; A reward function based on the global dissipation time is established, as shown in the following formula: In the formula, The reward function for the rapid evacuation signal control decision model is the sum of the weights of the total predicted number of queued vehicles at all entrances at the next time t+1 and the absolute value of the total pressure gradient of all entrances at the current time t. and These are the weighting coefficients.

7. The method for rapid evacuation control of intersection queues according to claim 1, characterized in that, The meta-reinforcement learning model for rapid queuing and evacuation at intersections is meta-trained to obtain optimal initial meta-parameters. Specifically, a meta-learning algorithm is used for two-stage training: inner loop adaptation and outer loop meta-optimization.

8. The method for rapid evacuation control of intersection queues according to claim 7, characterized in that, The specific training process is as follows: Initialize meta parameters ; From multiple different traffic congestion tasks A batch of sampling tasks Where N is the number of tasks; for each task The data is divided into a support set and a query set; For each task Execute inner loop adaptation: with the current meta initial parameters Starting with the support set data, the application calculates the reward function and policy gradient, performs gradient descent on the policy gradient, and obtains the adaptation parameters for the task. ; For all tasks, perform outer loop optimization: based on the adaptation parameters for each task. The loss value for each task is calculated using the query set data. The total loss function is obtained by summing the loss values ​​of all tasks. The primary optimization objective is to minimize the total loss function, and the primary initial parameters are updated using gradient descent. ; Until the initial parameters Convergence is achieved, and the optimal initial parameters of the trained system are obtained. and signal control strategy .

9. The method for rapid evacuation control of intersection queues according to claim 8, characterized in that, S6 executes traffic signal control decisions and generates the next execution phase, specifically: S6.1 The trained optimal initial parameters Load the rapid evacuation model for queuing at intersections into a meta-reinforcement learning model; S6.2 Collect traffic data at the intersection as a support set for the new task, and adjust the model parameters. Perform gradient updates to generate the adapted optimal initial meta-parameters. ; S6.3 uses an adapted intersection queuing fast evacuation meta-reinforcement learning model to perform real-time signal control at the intersection, generating the optimal signal control strategy based on the current model input and reward function. The optimal phase action is output as the next phase.

Images