An urban arterial adaptive signal control method combining transformer and meta-reinforcement learning

Abstract

The application discloses a kind of urban arterial road adaptive signal control methods combined with Transform and meta reinforcement learning.The application proposes the multi-agent cooperation structure of classification training dispersion decision, defines green wave waiting vehicle, designs the Transform module capable of capturing potential green wave demand by historical state, designs a reward function in establishing the Markov decision model of single agent, which balances the benefits of coordination direction and the overall benefits of intersection, proposes a double-layer meta learning Bi-MAML framework, designs a multi-agent PPO algorithm training process based on Bi-MAML, and realizes the adaptive signal control of urban arterial road.The arterial road adaptive signal control method proposed by the application has the advantages of capturing potential green wave demand, balancing the benefits of coordination direction and the overall benefits of intersection, reducing training cost and ensuring the relevance and effectiveness of agent, solving the poor model transferability of multiple intersections, etc., effectively solving the problem of high training cost of urban arterial road multiple intersections while realizing adaptive signal control.

Description

Technical Field

[0001] This invention belongs to the technical field of traffic signal control, and in particular to an adaptive signal control method for urban arterial roads that combines Transformer and meta-reinforcement learning. Background Technology

[0002] Green wave coordination on arterial roads is a crucial technical means in urban traffic signal control. By coordinating signal timing schemes at various intersections, vehicles can pass through multiple intersections on the coordinated direction of the arterial road at a certain speed without stopping, effectively reducing the number of stops, lowering fuel consumption and carbon emissions, and improving the traffic efficiency of the arterial road. Traditional green wave control methods for arterial roads include model-based methods, numerical methods, and graphical methods, but all rely on fixed signal timing schemes. When sudden situations occur on road sections causing instantaneous changes in traffic conditions, the coordination capability will be greatly weakened. To overcome the limitations of traditional green wave control, intelligent traffic control methods based on deep reinforcement learning have gradually become a research hotspot in recent years. These methods can make intelligent decisions in complex environments and achieve adaptive coordinated control.

[0003] In adaptive signal control research, retaining the common period and phase difference of each coordinated intersection in traditional green wave theory makes it difficult to synchronously adjust the schemes for all intersections. Furthermore, when each intersection is trained independently by a separate agent, the training cost becomes extremely high as the road network size increases. To reduce training time, some studies have considered centralized training for all intersections. The first approach uses a joint state space and action space; however, this method suffers from dimensionality explosion as the number of agents increases. The second approach allows agents at each intersection to share training samples and network parameters, but this is only suitable for homogeneous intersection groups and not for intersections with significantly different shape characteristics. Regarding traffic efficiency, traditional green wave coordination methods only consider the green wave benefits of arterial roads, without considering the balance between arterial and local roads or the overall efficiency of the intersections. Therefore, how to break the constraints of common cycle and phase difference to achieve synchronous update of signal schemes at various coordinated intersections, capture real-time vehicle green wave demand to adjust the scheme, pay attention to the fairness of branch roads while pursuing the benefits of coordinated direction, conduct group training and decentralized decision-making for intersections, and establish a training architecture with scalability and transferability has important research value and practical significance. Summary of the Invention

[0004] This invention provides an adaptive signal control method for urban arterial roads that combines Transformer and meta-reinforcement learning.

[0005] This invention addresses the shortcomings of existing traffic arterial coordination methods, such as their inability to achieve adaptive control that tracks changes in traffic flow, the inability of existing multi-intersection training methods to simultaneously address the issues of multi-agent scalability and dimensionality explosion, and the lack of consideration for both coordination direction benefits and overall intersection benefits. It employs Transformer to extract historical state features, thereby capturing potential green wave waiting vehicle demands. A multi-objective reward function considering both green wave efficiency and overall intersection traffic efficiency is designed. A learning mode of multi-intersection group training and decentralized decision-making is constructed. A bi-level meta-learning (Bi-MAML) algorithm learning framework is proposed, and a multi-agent PPO algorithm training process based on the Bi-MAML framework is designed.

[0006] This invention is achieved through the following technical solution:

[0007] An adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning includes the following steps:

[0008] S1. Determine the coordination direction of the main road, usually the direction with high traffic volume;

[0009] S2. Group the intersections by category;

[0010] S2-1. Determine the shape type of each intersection and classify intersections according to their shape into pedestrian crossing intersections, three-way intersections (T-shaped intersections), four-way intersections (cross-shaped intersections), five-way intersections, etc.

[0011] S2-2. Determine the number and location of adjacent intersections for each intersection, and classify the adjacent intersections of each intersection into two categories according to their location: adjacent intersections within the coordination area and adjacent intersections outside the coordination area.

[0012] If an adjacent intersection is located on the same direction as the main road, it is considered an adjacent intersection within the coordination scope; if an adjacent intersection is not located on the same direction as the main road, it is considered an adjacent intersection outside the coordination scope. By clarifying the number and location type of adjacent intersections for each intersection, the intersections can be further subdivided.

[0013] S2-3. Determine the grouping of intersections; classify intersections according to their shape type, the number of adjacent intersections, and their location type. Only intersections with the same shape type, the same number of adjacent intersections, and the same location type can be grouped together.

[0014] S3. Determine the detection range for each intersection. The detection range of an intersection includes the distance from the stop line to the start of the road segment in each approach direction. A minimum value (length of the solid line segment in the approach direction) and a maximum value (total road segment length in the approach direction) are set for this detection length. Based on the results of grouping intersections in step S2, set the same detection range for intersections in the same group. detect ;

[0015] S4. Define green wave waiting vehicles: Vehicles that proceed straight through the green light from the upstream intersection within the coordination range in the coordination direction are defined as green wave waiting vehicles when they arrive at the downstream intersection.

[0016] S5. Based on the Transformer design, a neural network structure capable of extracting historical state features of intersections is presented. The improved Transformer module designed in this invention includes an encoder, a decoder, and a final output layer; both the encoder and decoder consist of n... layer The network consists of layers, each employing the same network structure. In each layer of the encoder, data is input to a multi-head attention mechanism, then undergoes a linear transformation via a feedforward layer before being output. In each layer of the decoder, the data only needs to undergo a linear transformation via a feedforward layer. After passing through the encoder and decoder, the output sequence data undergoes mean pooling to reduce the time step dimension to 1, followed by another linear layer to compress the data dimension to d. out ;

[0017] Specifically, for the multi-head attention mechanism of each encoder layer, the input data is a fixed-dimensional matrix X; each head is a self-attention mechanism, obtaining the query matrix Q through three linear transformations. i Key matrix K i Sum matrix V i Define the weight matrices for the three linear transformations as follows: and The result Z of the multi-head attention mechanism is the result Z of all heads. i After splicing, it is obtained through linear transformation, and the calculation formula is shown in equation (1);

[0018] Z = Concat(Z1, Z2, ... Z) h W O (1)

[0019]

[0020] Where h is the number of long positions; W O It is the linear transformation matrix of the output; d k It is the dimension of the key matrix;

[0021] S6. For each intersection, establish a single-agent Markov decision model;

[0022] S6-1. The designed action is to select the next execution phase at the local intersection. The action set is represented as A = {a1, a2, ..., a...} k}, where k represents the number of selectable phases; the selected phase will execute a fixed-interval green light time Δt; within one signal cycle, the total green light time of a single phase is subject to a minimum green light time t. G_min and maximum green light time t G_max Constraints; when a phase switch occurs, a yellow light time t needs to be executed before the new phase is executed. Y Transition;

[0023] S6-2, The design state includes the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector of the local intersection;

[0024] Before recording the vehicle density matrix and vehicle speed matrix, the lanes at the local intersection are divided into multiple lane groups according to the traffic flow direction; then each lane group is further divided into blocks, with a block length of l. cell Set as the distance between the front ends h s The value must be an integer multiple of the specified value, and the block area must not exceed the detection range. detect Count the number of vehicles and the average speed of vehicles in each lane block, and then normalize them.

[0025] The historical action matrix consists of the historical actions of adjacent intersections and local intersections. The action vector is defined as a one-hot vector of size k. Specifically, assuming the historical data acquisition step size is τ, at time t, the action vectors of local intersections at times t-Δt·m, m=0,1,……,……τ are obtained, and the action vectors of adjacent intersections within the coordination range at times t-Δt·m, m=0,1,……,……τ are also obtained. The action vector at time t, where t o→d Indicates from adjacent intersection I o Driving to local intersection I d The travel time required to pass through without stopping. Indicates the calculation of t o→d / Δt is rounded up; the action vectors of the two adjacent intersections and the local intersection are combined into a matrix, where, when there are no adjacent intersections, all action vectors represented by them are filled with 0 values;

[0026] The historical state feature vector is obtained based on the historical vehicle density matrix of the local intersection's coordinated direction and the historical vehicle density matrix of the adjacent intersection's coordinated direction. Specifically, assuming the historical data acquisition step size is τ, at time t, the vehicle density matrix of the local intersection at times t-Δt·m, m=0,1,……,τ is obtained for the coordinated direction, and the vehicle density matrix of the adjacent intersections within the coordinated range at times t-Δt·m, m=0,1,……,τ is obtained for the coordinated direction. The vehicle density matrix of the coordinated direction at time t, where t o→d Indicates from adjacent intersection I o Driving to local intersection I d The travel time required to pass through without stopping. Indicates the calculation of t o→d / Δt is rounded up; the historical vehicle density matrix of the local intersection coordination direction and the historical vehicle density matrix of the adjacent intersection coordination direction are concatenated together and input into the Transformer module for feature extraction, and the historical state feature vector is output.

[0027] Finally, the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector are concatenated to form the final input state vector.

[0028] S6-3. For each intersection, the reward function is calculated independently based on the traffic conditions of that intersection. The calculation formula of the reward function is shown in equation (3).

[0029]

[0030] Among them, R t This represents the reward value at time t; The waiting time reward value represents the waiting time of vehicles waiting in the green wave at time t. The calculation formula is shown in equation (4). The waiting time reward value of other vehicles at time t is represented by the formula shown in equation (5); ρ is a coefficient used to adjust the ratio of the waiting time reward value of other vehicles to the total reward.

[0031]

[0032] Among them, w m This represents the cumulative waiting time for vehicle m. This represents the number of vehicles waiting in the green wave at time t. This represents the number of other non-green wave vehicles waiting to proceed at time t;

[0033] S7. Construct a multi-agent collaborative structure for classification training and decentralized decision-making. Under this structure, intersections in the same group are trained centrally, and all intersections make decentralized decisions. Specifically, agents in the same group of intersections share experience samples for iterative training and share network parameters. However, when making decisions, each intersection makes its own independent decision based on its current state information.

[0034] S8. Construct a two-layer meta-learning Bi-MAML framework. Under this framework, the intersection of a certain group is set as the meta-learner, and other agents are regarded as individual learners. The meta-learner includes all intersections within the corresponding group, and these intersections share the network parameters of the meta-learner. The first layer of meta-learning includes meta-training and fine-tuning, where both meta-training and fine-tuning include inner layer updates and outer layer updates. The second layer of meta-learning only has outer layer updates.

[0035] When the meta-learner has not yet reached the set meta-training round E during training meta Meta-training is performed, focusing solely on the meta-learner. Each intersection corresponding to the meta-learner is treated as an independent task, and tasks are randomly sampled to obtain different batches. For each batch of tasks, corresponding samples are obtained and divided into two categories, named support set samples and query set samples, respectively. For each batch of tasks, inner and outer layer updates are performed. A network replica is created by copying the model parameters θ of the meta-learner, and inner layer updates are performed on this replica. In the first update, the loss function L is calculated using the support set samples. supp (θ), and obtain θ′ by updating the parameters of the network replica using formula (6). Subsequently, the network replica parameter θ′ will be iteratively updated multiple times on the support set.

[0036]

[0037] Where α is the learning rate of the inner layer update. It is the gradient of the parameter θ;

[0038] In the outer layer update, the loss function L is calculated on the updated network replica using query set samples. query (θ′); L is calculated on the query set samples for each batch task. query (θ′) is used to calculate the mean value of L(θ′), and the original network is trained and updated using the gradient descent method. The parameter update formula is shown in equation (7).

[0039]

[0040] Where β is the learning rate of the outer layer update;

[0041] When the meta-learner reaches the set meta-training round E during training meta At that time, fine-tuning is performed; in fine-tuning, an inner layer update and an outer layer update are required for each of the intelligent agents, including the meta-learner and each individual learner.

[0042] After the meta-learner and all individual learners have been fine-tuned once, the second layer of learning is carried out. For all agents, including the meta-learner and all individual learners, each agent samples from the experience pool without distinguishing between support set samples and query set samples, and updates the network of each learner using gradient descent.

[0043] S9. Train the multi-agent PPO algorithm model based on the Bi-MAML framework of bi-layer meta-learning;

[0044] S9-1. Design the Actor network and Critic network structures, and initialize the network parameters;

[0045] S9-2. Initialize the road network environment and signaling scheme;

[0046] S9-3, Obtain the current state s of each intersection. t Each input is fed into its respective Actor network, and then each outputs its chosen action 'a'. t ;

[0047] S9-4. Control the operation of the signal controllers selected for each intersection;

[0048] S9-5, Each intersection obtains its own reward r t The state is input into the Critic network and the value function is output. Then the state, action, value function and reward are stored in the experience pool of each intersection as tuples.

[0049] S9-6. Determine whether the number of samples meets the sampling conditions. If the sampling conditions are met, proceed to step S9-7; otherwise, skip to step S9-13.

[0050] S9-7. Determine if the number of completed training rounds has reached the set meta-training round E. meta If the condition has been met, proceed to step S9-9; otherwise, proceed to step S9-8.

[0051] S9-8. Perform the first layer training, the specific steps are as follows:

[0052] S9-8-1. Treat each intersection as a task and perform multiple batches of random sampling on the intersections using meta-learners. Each batch contains several intersections (i.e., several tasks).

[0053] S9-8-2. Extract samples from the experience pool at each intersection and divide the samples into two parts, which will serve as the support set and query set for this task, respectively.

[0054] S9-8-3. Perform the first layer of meta-training on the meta-learner, where the outer layer update realizes the updating of the parameters of the Actor network and Critic network of the meta-learner;

[0055] S9-8-4. Copy the network parameters of the meta-learner to other agents, and then proceed to step S9-12;

[0056] S9-9. Determine whether fine-tuning has been performed. If fine-tuning has been performed, proceed to step S9-11; if fine-tuning has not been performed, proceed to step S9-10.

[0057] S9-10. Perform the first layer of fine-tuning, the specific steps are as follows:

[0058] S9-10-1. Each experience pool draws multiple batches of samples as support sets and query sets.

[0059] S9-10-2. Each agent performs first-layer fine-tuning based on its own samples, and updates its own Actor network and Critic network parameters through outer layer updates, and then proceeds to step S9-12.

[0060] S9-11. Proceed to the second layer of learning, with the following specific steps:

[0061] S9-11-1. Each agent draws multiple batches of samples from its own experience pool and calculates the loss functions of the Actor network and the Critic network according to Equations (8) and (9), respectively.

[0062] L actor (θ1)=E t [min(p t (θ1)A t ,clip(p t (θ1), 1-ε, 1+ε)A t (8)

[0063]

[0064] Where θ1 represents the Actor network parameters; θ2 represents the Critic network parameters; p t (θ1) is in state s t Next, based on the current strategy Compared to the old strategy Select action a respectively t The ratio of the probabilities; clip(p) t (θ1), 1-ε, 1+ε) is a set of p t (θ1) is a function restricted to the range [1-ε, 1+ε]. Indicates the state s tThe current value function; V t target A represents the objective value function; t The dominant function is represented by γ, the reward reduction factor is γ, and the termination time is T. This indicates that the old value network represents state s. T The estimated value;

[0065] S9-11-2. Each agent updates its own Actor network and Critic network parameters according to the calculated loss function, and then proceeds to step S9-12.

[0066] S9-12. Clear the experience pool and proceed to step S9-13.

[0067] S9-13. Determine whether the end time of this round has been reached. If the end time has been reached, proceed to step S9-14; if the end time has not been reached, return to step S9-3.

[0068] S9-14. Determine whether all rounds of iteration have been completed. If all rounds of iteration have been completed, end the training. If not, return to step S9-2.

[0069] S10. Using the trained agent models, implement adaptive signal control for urban arterial roads.

[0070] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0071] (1) Extract the historical state features of local and adjacent intersections through Transformer, capture potential green wave demand, and dynamically adjust the signal phase scheme.

[0072] (2) Group the intersections according to their different characteristics such as shape, size, and lane organization. Use shared samples and network parameters for intersections in the same group to reduce training costs. At the same time, avoid the problem of low adaptability and weak targeting caused by using the same network parameters for non-homogeneous intersections.

[0073] (3) The proposed Bi-MAML framework can solve the problem of poor mobility between different intersections in a large-scale road network. By training meta-learners separately in the first layer and training each agent independently in the second layer, the convergence speed is accelerated in the early stage of training and the applicability of each agent is enhanced in the later stage of training. Attached Figure Description

[0074] Figure 1 This is a flowchart of the adaptive signal control method for urban arterial roads that combines Transformer and meta-reinforcement learning according to an embodiment of the present invention.

[0075] Figure 2 This is a schematic diagram of the main road according to an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of the selectable signal phases in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of lane grouping and segmentation at an intersection according to an embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram of intelligent agent classification according to an embodiment of the present invention;

[0079] Figure 6 This is a flowchart of the meta-training process in Bi-MAML, a two-layer meta-learning method according to an embodiment of the present invention.

[0080] Figure 7 This is a flowchart of the fine-tuning process in Bi-MAML, a two-layer meta-learning method, according to an embodiment of the present invention.

[0081] Figure 8 This is a flowchart of the training process of the multi-agent PPO algorithm based on the Bi-MAML framework in this embodiment of the invention. Detailed Implementation

[0082] The solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0083] Please see Figure 1 This embodiment provides an adaptive signal control method for urban arterial roads that combines Transformer and meta-reinforcement learning, including the following steps:

[0084] S1. Determine the coordination direction of the main road, which is usually the direction with high traffic volume; in one embodiment, the east-west direction is used as the coordination direction.

[0085] S2. Group the intersections by category;

[0086] S2-1. Determine the shape type of each intersection and classify intersections according to their shape into pedestrian crossing intersections, three-way intersections (T-shaped intersections), four-way intersections (cross-shaped intersections), five-way intersections, etc.

[0087] S2-2. Determine the number and location of adjacent intersections for each intersection, and classify the adjacent intersections of each intersection into two categories according to their location: adjacent intersections within the coordination area and adjacent intersections outside the coordination area.

[0088] If an adjacent intersection is located on the same direction as the main road, it is considered an adjacent intersection within the coordination scope; if an adjacent intersection is not located on the same direction as the main road, it is considered an adjacent intersection outside the coordination scope. By clarifying the number and location type of adjacent intersections for each intersection, the intersections can be further subdivided.

[0089] S2-3. Determine the grouping of intersections; classify intersections according to their shape type, the number of adjacent intersections, and their location type. Only intersections with the same shape type, the same number of adjacent intersections, and the same location type can be grouped together.

[0090] Please see Figure 2 For example, taking a main road with eight intersections as an example, intersection I1 is assigned to Agent1 as an end T-junction, intersections I2 and I3 are assigned to Agent2 as a middle T-junction, intersections I4, I6 and I7 are assigned to Agent3 as a middle crossroads, intersection I5 is assigned to Agent4 as a pedestrian crossing, and intersection I8 is assigned to Agent5 as an end crossroads.

[0091] S3. Determine the detection range for each intersection. The detection range of an intersection includes the distance from the stop line to the start of the road segment in each approach direction. A minimum value (length of the solid line segment in the approach direction) and a maximum value (total road segment length in the approach direction) are set for this detection length. Based on the results of grouping intersections in step S2, set the same detection range for intersections in the same group. detect In one embodiment, the same detection range is set for all agents (including intersections of different groups). detect =100m;

[0092] S4. Define green wave waiting vehicles; vehicles that proceed straight through a green light from the upstream intersection within the coordination range in the coordination direction are defined as green wave waiting vehicles when they arrive at the downstream intersection; please refer to Figure 2 For example, a vehicle that goes straight through the west entrance of intersection I1 and arrives at intersection I2 is identified as a green wave waiting vehicle. Conversely, a vehicle that turns right through the south entrance of intersection I1 and arrives at intersection I2 is identified as a non-green wave waiting vehicle.

[0093] S5. Based on the Transformer design, a neural network structure capable of extracting historical state features of intersections is presented. The improved Transformer module designed in this invention includes an encoder, a decoder, and a final output layer; both the encoder and decoder consist of n... layerThe network consists of layers, each employing the same network structure. In each layer of the encoder, data is input to a multi-head attention mechanism, then undergoes a linear transformation via a feedforward layer before being output. In each layer of the decoder, the data only needs to undergo a linear transformation via a feedforward layer. After passing through the encoder and decoder, the output sequence data undergoes mean pooling to reduce the time step dimension to 1, followed by another linear layer to compress the data dimension to d. out ;

[0094] Specifically, for the multi-head attention mechanism of each encoder layer, the input data is a fixed-dimensional matrix X; each head is a self-attention mechanism, obtaining the query matrix Q through three linear transformations. i Key matrix K i Sum matrix V i Define the weight matrices for the three linear transformations as follows: and The result Z of the multi-head attention mechanism is the result Z of all heads. i After splicing, it is obtained through linear transformation, and the calculation formula is shown in equation (1);

[0095] Z = Concat(Z1, Z2, ... Z) h W O (1)

[0096]

[0097] Where h is the number of long positions; W O It is the linear transformation matrix of the output; d k It is the dimension of the key matrix;

[0098] In one embodiment, n is set layer =6, meaning the data needs to go through 6 layers of encoders and decoders; set d out =16, meaning the final output vector size from the Transformer module is 16; h=8, meaning an 8-head self-attention mechanism is used; d k =d v =100, meaning that the dimensions of both the key matrix and the value matrix are 100;

[0099] S6. For each intersection, establish a single-agent Markov decision model;

[0100] S6-1. The designed action is to select the next execution phase at the local intersection. The action set is represented as A = {a1, a2, ..., a...} k}, where k represents the number of selectable phases; the selected phase will execute a fixed-interval green light time Δt; within one signal cycle, the total green light time of a single phase is subject to a minimum green light time t. G_minand maximum green light time t G_max Constraints; when a phase switch occurs, a yellow light time t needs to be executed before the new phase is executed. Y Transition is underway; please refer to [link / reference]. Figure 3 In one example, the total number of selectable phases is k = 8. Each action represents one selectable phase. The selectable phases differ for different intersections. Invalid phases need to be masked when making action decisions, but the action space size of all agents must still be consistent. In one embodiment, a fixed green light interval Δt = 5s is set, and the minimum green light time t G_min =15s, maximum green light time t G_max =60s, yellow light duration t Y =5s;

[0101] S6-2, The design state includes the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector of the local intersection;

[0102] Before recording the vehicle density matrix and vehicle speed matrix, the lanes at the local intersection are divided into multiple lane groups according to the traffic flow direction; then each lane group is further divided into blocks, with a block length of l. cell Set as the distance between the front ends h s The value must be an integer multiple of the specified value, and the block area must not exceed the detection range. detect Count the number of vehicles and average speed within each lane block, and then normalize the data; please refer to [link / reference]. Figure 4 One example is that the lanes are divided into eight groups according to traffic flow direction, such as left turn at the east entrance, straight ahead at the east entrance, and left turn at the west entrance. The diagram shows an example of the lane group division for left turn at the west entrance and straight ahead at the west entrance. When dividing the lanes, l is set... cell =h s =7.5m;

[0103] The historical action matrix consists of the historical actions of adjacent intersections and local intersections. The action vector is defined as a one-hot vector of size k. Specifically, assuming the historical data acquisition step size is τ, at time t, the action vectors of local intersections at times t-Δt·m, m=0,1,……,……τ are obtained, and the action vectors of adjacent intersections within the coordination range at times t-Δt·m, m=0,1,……,……τ are also obtained. The action vector at time t, where t o→d Indicates from adjacent intersection I o Driving to local intersection I d The travel time required to pass through without stopping. Indicates the calculation of t o→d / Δt is rounded up; the action vectors of the two adjacent intersections and the local intersection are combined into a matrix, where, when there are no adjacent intersections, all action vectors represented by them are filled with 0 values;

[0104] The historical state feature vector is obtained based on the historical vehicle density matrix of the local intersection's coordinated direction and the historical vehicle density matrix of the adjacent intersection's coordinated direction. Specifically, assuming the historical data acquisition step size is τ, at time t, the vehicle density matrix of the local intersection at times t-Δt·m, m=0,1,……,τ is obtained for the coordinated direction, and the vehicle density matrix of the adjacent intersections within the coordinated range at times t-Δt·m, m=0,1,……,τ is obtained for the coordinated direction. The vehicle density matrix of the coordinated direction at time t, where t o→d Indicates from adjacent intersection I o Driving to local intersection I d The travel time required to pass through without stopping. Indicates the calculation of t o→d / Δt is rounded up; the historical vehicle density matrix of the local intersection coordination direction and the historical vehicle density matrix of the adjacent intersection coordination direction are concatenated together and input into the Transformer module for feature extraction, and the historical state feature vector is output.

[0105] Please see Figure 2 As an example, given τ = 5 and Δt = 5s, with intersection I1 as the local intersection, the travel time t required to travel from upstream intersection I2 to local intersection I1 is... 2→1 =28s, then Obtain the historical action vectors and east-west historical state matrices of I1 at state update times t, t-5, t-10, t-15, and t-20, respectively; and the historical action vectors and east-west historical state matrices of I2 at state update times t-30, t-35, t-40, t-45, and t-50, respectively. Since I1 has only one adjacent intersection, fill the relevant data of other non-existent adjacent intersections with 0 values. Concatenate all historical action vectors into a matrix, and concatenate all historical state matrices into a new matrix, then input it into a Transformer to transform it into a d-dimensional matrix. out =16 historical state feature vectors;

[0106] Finally, the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector are concatenated to form the final input state vector.

[0107] S6-3. For each intersection, the reward function is calculated independently based on the traffic conditions of that intersection. The calculation formula of the reward function is shown in equation (3).

[0108]

[0109] Among them, R t This represents the reward value at time t; The waiting time reward value represents the waiting time of vehicles waiting in the green wave at time t. The calculation formula is shown in equation (4). The waiting time reward value of other vehicles at time t is represented by the formula shown in equation (5); ρ is a coefficient used to adjust the ratio of the waiting time reward value of other vehicles to the total reward.

[0110]

[0111] Among them, w m This represents the cumulative waiting time for vehicle m. This represents the number of vehicles waiting in the green wave at time t. This represents the number of other non-green wave vehicles waiting to proceed at time t; in one embodiment, ρ = 0.9;

[0112] S7. Construct a multi-agent collaborative structure for classification training and decentralized decision-making. Under this structure, intersections in the same group are trained centrally, and all intersections make decentralized decisions. Specifically, agents in the same group of intersections share experience samples for iterative training and share network parameters. However, when making decisions, each intersection makes its own independent decision based on its current state information.

[0113] S8. Construct a two-layer meta-learning Bi-MAML framework. Within this framework, a crossroads within a specific group are designated as meta-learners, and other agents are treated as individual learners. The meta-learner includes all crossroads within the corresponding group, and these crossroads share the network parameters of the meta-learner. The first layer of meta-learning includes meta-training and fine-tuning, both of which involve inner and outer layer updates. The second layer of meta-learning only involves outer layer updates. Please refer to [link / reference]. Figure 5 In one example, intersections I4, I6, and I7 corresponding to Agent1 are designated as meta-learners, sharing samples and network parameters among themselves; other agents are set as individual learners; and the meta-training round E is set. meta =50;

[0114] When the meta-learner has not yet reached the set meta-training round E during training meta At that time, meta-training will be conducted; this training is only for meta-learners. Please refer to [link / reference]. Figure 6This demonstrates the meta-training process. Each intersection corresponding to the meta-learner is treated as an independent task. Random sampling is performed on these tasks to obtain different batches. For each batch, corresponding samples are acquired and divided into two categories: support set samples and query set samples. For each batch, inner and outer layer updates are performed. A network replica is created by copying the model parameters θ of the meta-learner. Inner layer updates are performed on this replica. In the first update, the loss function L is calculated using the support set samples. supp (θ), and obtain θ′ by updating the parameters of the network replica using formula (6). Subsequently, the network replica parameter θ′ will be iteratively updated multiple times on the support set.

[0115]

[0116] Where α is the learning rate of the inner layer update. It is the gradient of the parameter θ; in one embodiment, α is set to 0.001;

[0117] In the outer layer update, the loss function L is calculated on the updated network replica using query set samples. query (θ′); L is calculated on the query set samples for each batch task. query (θ′) is used to calculate the mean value of L(θ′), and the original network is trained and updated using the gradient descent method. The parameter update formula is shown in equation (7).

[0118]

[0119] Where β is the learning rate of the outer layer update; in one embodiment, β is set to 0.0001;

[0120] When the meta-learner reaches the set meta-training round E during training meta Make fine adjustments as needed; please refer to [link / reference]. Figure 7 This demonstrates the fine-tuning process; in fine-tuning, it is necessary to perform an inner layer update and an outer layer update for each agent, including the meta-learner and all individual learners.

[0121] After the meta-learner and all individual learners have been fine-tuned once, the second layer of learning is carried out. For all agents, including the meta-learner and all individual learners, each agent samples from the experience pool without distinguishing between support set samples and query set samples, and updates the network of each learner using gradient descent.

[0122] S9. Train the multi-agent PPO algorithm model based on the Bi-MAML framework; please refer to [link / reference]. Figure 8 The training process is as follows:

[0123] S9-1. Design the Actor network and Critic network structures, and initialize the network parameters;

[0124] S9-2. Initialize the road network environment and signaling scheme;

[0125] S9-3, Obtain the current state s of each intersection. t Each input is fed into its respective Actor network, and then each outputs its chosen action 'a'. t ;

[0126] S9-4. Control the operation of the signal controllers selected for each intersection;

[0127] S9-5, Each intersection obtains its own reward r t The state is input into the Critic network and the value function is output. Then the state, action, value function and reward are stored in the experience pool of each intersection as tuples.

[0128] S9-6. Determine whether the number of samples meets the sampling conditions. If the sampling conditions are met, proceed to step S9-7; otherwise, skip to step S9-13.

[0129] S9-7. Determine if the number of completed training rounds has reached the set meta-training round E. meta If the condition has been met, proceed to step S9-9; otherwise, proceed to step S9-8.

[0130] S9-8. Perform the first layer training, the specific steps are as follows:

[0131] S9-8-1. Treat each intersection as a task and perform multiple batches of random sampling on the intersections using meta-learners. Each batch contains several intersections (i.e., several tasks).

[0132] S9-8-2. Extract samples from the experience pool at each intersection and divide the samples into two parts, which will serve as the support set and query set for this task, respectively.

[0133] S9-8-3. Perform the first layer of meta-training on the meta-learner, where the outer layer update realizes the updating of the parameters of the Actor network and Critic network of the meta-learner;

[0134] S9-8-4. Copy the network parameters of the meta-learner to other agents, and then proceed to step S9-12;

[0135] S9-9. Determine whether fine-tuning has been performed. If fine-tuning has been performed, proceed to step S9-11; if fine-tuning has not been performed, proceed to step S9-10.

[0136] S9-10. Perform the first layer of fine-tuning, the specific steps are as follows:

[0137] S9-10-1. Each experience pool draws multiple batches of samples as support sets and query sets;

[0138] S9-10-2. Each agent performs first-layer fine-tuning based on its own samples, and updates its own Actor network and Critic network parameters through outer layer updates, and then proceeds to step S9-12.

[0139] S9-11. Proceed to the second layer of learning, with the following specific steps:

[0140] S9-11-1. Each agent draws multiple batches of samples from its own experience pool and calculates the loss functions of the Actor network and the Critic network according to Equations (8) and (9), respectively.

[0141]

[0142] Where θ1 represents the Actor network parameters; θ2 represents the Critic network parameters; p t (θ1) is in state s t Next, based on the current strategy Compared to the old strategy Select action a respectively t The ratio of the probabilities; clip(p) t (θ1), 1-ε, 1+ε) is a set of p t (θ1) is a function restricted to the range [1-ε, 1+ε]. Indicates the state s t The current value function; V t target A represents the objective value function; t The dominant function is represented by γ, the reward reduction factor is γ, and the termination time is T. This indicates that the old value network represents state s. T The estimated value; in one embodiment, ε = 0.2, γ = 0.98, T = 3600s;

[0143] S9-11-2. Each agent updates its own Actor network and Critic network parameters according to the calculated loss function, and then proceeds to step S9-12.

[0144] S9-12. Clear the experience pool and proceed to step S9-13.

[0145] S9-13. Determine whether the end time of this round has been reached. If the end time has been reached, proceed to step S9-14; if the end time has not been reached, return to step S9-3.

[0146] S9-14. Determine whether all rounds of iteration have been completed. If all rounds of iteration have been completed, end the training. If not, return to step S9-2.

[0147] S10. Using the trained agent models, implement adaptive signal control for urban arterial roads.

[0148] When dealing with time-varying traffic flow in real traffic environments, information such as vehicle location, vehicle speed, and traffic light status is acquired through Internet of Things (IoT) technology. This information is then converted into state data, which is input to each trained agent and outputs decision-making actions to achieve adaptive signal control on urban arterial roads.

[0149] As described above, the present invention can be implemented well.

[0150] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning, characterized in that, Includes the following steps: S1. Determine the coordination direction of the main road; S2. Group the intersections by category; S3. Determine the detection range of each intersection. The detection range of an intersection includes the distance from the stop line to the start of the road segment in each direction of approach. S4. Define green wave waiting vehicles: Vehicles that proceed straight through the green light from the upstream intersection within the coordination range in the coordination direction are defined as green wave waiting vehicles when they arrive at the downstream intersection. S5. Based on Transformer, a neural network structure capable of extracting historical state features of intersections is designed. The improved Transformer module includes an encoder, a decoder, and a final output layer. S6. For each intersection, establish a single-agent Markov decision model; S7. Construct a multi-agent collaborative structure for classification training and decentralized decision-making. Under this multi-agent collaborative structure, intersections of the same group are trained centrally, and all intersections make decentralized decision-making actions. S8. Construct a two-layer meta-learning Bi-MAML framework. Under this framework, the intersection of a certain group is set as the meta-learner, and other agents are regarded as individual learners. The first layer of meta-learning includes meta-training and fine-tuning. Both meta-training and fine-tuning include inner layer updates and outer layer updates. The second layer of meta-learning only has outer layer updates. S9. Train the multi-agent PPO algorithm model based on the Bi-MAML framework of bi-layer meta-learning; S10. Using the trained agent models, implement adaptive signal control for urban arterial roads. Step S6, establishing the single-agent Markov decision model, specifically includes the following sub-steps: S6-1, The design action is to select the next execution phase at the local intersection; S6-2, The design state includes the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector of the local intersection; S6-3. For each intersection, the reward function is calculated independently based on the traffic conditions of that intersection. The calculation formula of the reward function is shown in Equation (1). Among them, R t This represents the reward value at time t; The waiting time reward value represents the waiting time of vehicles waiting in the green wave at time t. The calculation formula is shown in Equation (2). The waiting time reward value of other vehicles at time t is represented by the formula shown in equation (3); ρ is a coefficient used to adjust the ratio of the waiting time reward value of other vehicles to the total reward. Among them, w m This represents the cumulative waiting time for vehicle m. This represents the number of vehicles waiting in the green wave at time t. This represents the number of other non-green wave vehicles waiting to proceed at time t; Step S8, when the meta-learner has not yet reached the set meta-training round E during training. meta Meta-training is performed, focusing solely on the meta-learner. Each intersection corresponding to the meta-learner is treated as an independent task, and tasks are randomly sampled to obtain different batches. For each batch of tasks, corresponding samples are obtained and divided into two categories, named support set samples and query set samples, respectively. For each batch of tasks, inner and outer layer updates are performed. A network replica is created by copying the model parameters θ of the meta-learner, and inner layer updates are performed on this replica. In the first update, the loss function L is calculated using the support set samples. supp (θ), and obtain θ′ by updating the parameters of the network replica using formula (4). Subsequently, the network replica parameter θ′ will be iteratively updated multiple times on the support set. θ′=θ-α·▽ θ L supp (i) (4) Where α is the learning rate of the inner layer update, ▽ θ It is the gradient of the parameter θ; In the outer layer update, the loss function L is calculated on the updated network replica using query set samples. query (θ′); L is calculated on the query set samples for each batch task. query (θ′) is used to calculate the mean value of L(θ′), and the original network is trained and updated using the gradient descent method. The parameter update formula is shown in equation (5). θ←θ-β·▽ θ L(θ′) (5) Where β is the learning rate of the outer layer update; When the meta-learner reaches the set meta-training round E during training meta At that time, fine-tuning is performed; in fine-tuning, an inner layer update and an outer layer update are required for each of the intelligent agents, including the meta-learner and each individual learner. After the meta-learner and all individual learners have been fine-tuned once, the second layer of learning is carried out. For all agents, including the meta-learner and all individual learners, each agent samples from the experience pool without distinguishing between support set samples and query set samples, and updates its own network using gradient descent.

2. The urban arterial road adaptive signal control method combining Transformer and meta-reinforcement learning as described in claim 1, characterized in that, Step S2, grouping intersections by category, specifically includes the following sub-steps: S2-1. Determine the shape type of each intersection and classify intersections according to their shape into pedestrian crossing intersections, three-way intersections, four-way intersections, and five-way intersections; S2-2. Determine the number and location of adjacent intersections for each intersection, and classify the adjacent intersections of each intersection into two categories according to their location: adjacent intersections within the coordination area and adjacent intersections outside the coordination area. S2-3. Determine the grouping of intersections; classify intersections according to their shape type, the number of adjacent intersections, and their location type. Only intersections with the same shape type, the same number of adjacent intersections, and the same location type can be grouped together.

3. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 1, characterized in that, In step S5, both the encoder and decoder are composed of n layer The network consists of layers, each employing the same network structure. In each layer of the encoder, data is input to a multi-head attention mechanism, then undergoes a linear transformation via a feedforward layer before being output. In each layer of the decoder, the data only needs to undergo a linear transformation via a feedforward layer. After passing through the encoder and decoder, the output sequence data undergoes mean pooling to reduce the time step dimension to 1, followed by another linear layer to compress the data dimension to d. out .

4. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 1, characterized in that, In step S6-2, the design state includes the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector of the local intersection, specifically: Before recording the vehicle density matrix and vehicle speed matrix, the lanes at the local intersection are divided into multiple lane groups according to the traffic flow direction; then each lane group is further divided into blocks, with a block length of l. cell Set as the distance between the front ends h s The value must be an integer multiple of the specified value, and the block area must not exceed the detection range. detect Count the number of vehicles and the average speed of vehicles in each lane block, and then normalize them. The historical action matrix consists of the historical actions of adjacent intersections and local intersections. The action vector is defined as a one-hot vector of size k. The historical state feature vector is obtained based on the historical vehicle density matrix of the local intersection coordination direction and the historical vehicle density matrix of the adjacent intersection coordination direction. Finally, the vehicle density matrix, vehicle speed matrix, historical action matrix, and historical state feature vector are concatenated to form the final input state vector.

5. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 1, characterized in that, In step S9, training the multi-agent PPO algorithm model based on the Bi-MAML framework specifically includes the following sub-steps: S9-1. Design the Actor network and Critic network structures, and initialize the network parameters; S9-2. Initialize the road network environment and signaling scheme; S9-3, Obtain the current state s of each intersection. t Each input is fed into its respective Actor network, and then each outputs its chosen action 'a'. t ; S9-4. Control the operation of the signal controllers selected for each intersection; S9-5, Each intersection obtains its own reward r t The state is input into the Critic network and the value function is output. Then the state, action, value function and reward are stored in the experience pool of each intersection as tuples. S9-6. Determine whether the number of samples meets the sampling conditions. If the sampling conditions are met, proceed to step S9-7; otherwise, skip to step S9-13. S9-7. Determine if the number of completed training rounds has reached the set meta-training round E. meta If the condition has been met, proceed to step S9-9; otherwise, proceed to step S9-8. S9-8, Perform first-level training S9-9. Determine whether fine-tuning has been performed. If fine-tuning has been performed, proceed to step S9-11; if fine-tuning has not been performed, proceed to step S9-10. S9-10, Perform the first layer of fine-tuning; S9-11, Proceed to the second layer of learning; S9-12. Clear the experience pool and proceed to step S9-13. S9-13. Determine whether the end time of this round has been reached. If the end time has been reached, proceed to step S9-14; if the end time has not been reached, return to step S9-3. S9-14. Determine whether all rounds of iteration have been completed. If all rounds of iteration have been completed, end the training. If not, return to step S9-2.

6. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 5, characterized in that, In steps S9-8, the first-layer meta-training specifically includes the following sub-steps: S9-8-1 Treat each intersection as a task and perform multiple batches of random sampling on the intersections using meta-learners, with each batch containing several intersections; S9-8-2. Extract samples from the experience pool at each intersection and divide the samples into two parts, which will serve as the support set and query set for this task, respectively. S9-8-3. Perform the first layer of meta-training on the meta-learner, where the outer layer update realizes the updating of the parameters of the Actor network and Critic network of the meta-learner; S9-8-4. Copy the network parameters of the meta-learner to other agents, and then proceed to step S9-12.

7. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 5, characterized in that, In steps S9-10, the first-level fine-tuning specifically includes the following sub-steps: S9-10-1. Each experience pool draws multiple batches of samples as support sets and query sets. S9-10-2. Each agent performs first-layer fine-tuning based on its own samples, and updates its own Actor network and Critic network parameters through outer layer updates, and then proceeds to step S9-12.

8. The adaptive signal control method for urban arterial roads combining Transformer and meta-reinforcement learning as described in claim 5, characterized in that, In steps S9-11, the second layer of learning specifically includes the following sub-steps: S9-11-1. Each agent draws multiple batches of samples from its own experience pool and calculates the loss functions of the Actor network and the Critic network according to equations (6) and (7), respectively. L actor (θ1)=E t [min(p t (θ1)A t ,clip(p t (θ1),1-ε,1+ε)A t )] (6) Where θ1 represents the Actor network parameters; θ2 represents the Critic network parameters; p t (θ1) is in state s t Next, based on the current strategy Compared to the old strategy Select action a respectively t The ratio of the probabilities; clip(p) t (θ1), 1-ε, 1+ε is a set of p t (θ1) is a function restricted to the range [1-ε, 1+ε]. Indicates the state s t The current value function; V t target A represents the objective value function; t The dominant function is represented by γ, the reward reduction factor is γ, and the termination time is T. This indicates that the old value network represents state s. T The estimated value; S9-11-2. Each agent updates its own Actor network and Critic network parameters according to the calculated loss function, and then proceeds to step S9-12.

Images