Traffic signal control method based on multi-agent reinforcement learning and intersection importance

Abstract

The invention provides a traffic signal control method based on multi-agent reinforcement learning and intersection importance, and belongs to the technical field of traffic signal control. Constructing a CM-MG model based on the priority of the intersection, and setting a decision sequence for all agents in the traffic grid; and selecting an optimal action for each agent in the decision sequence through the GNSD-Light network, wherein the method comprises the following steps of: obtaining environment information through an observation presentation layer; historical action characteristics of each agent are extracted through a precursor action presentation layer, so that member agents understand and cooperate with a precursor decision; outputting the optimal action of each agent in sequence through a Q output layer; the observation presentation layer comprises the following steps: extracting spatial features through a simple spatial processing module based on relative position coding; and integrating neighborhood observation features through a residual image attention network. The traffic efficiency of the important intersection is improved; and thus, the traffic efficiency of global traffic is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic signal control, and in particular to a traffic signal control method based on multi-agent reinforcement learning and intersection importance. Background Art

[0002] Traditional traffic signal control (TSC) methods rely on fixed schedules and historical data and cannot adapt to real-time traffic fluctuations. Although the optimization control method of TSC is effective in modeling and signal adjustment, its parameters are easily affected by fluctuation factors.

[0003] Existing multi-agent reinforcement learning (MARL) in adaptive traffic signal control (ATSC) usually models the coordinated control of multiple intersections as a cooperative Markov game to optimize the average traffic efficiency of all intersections. Traffic efficiency is improved by dynamically optimizing signal timing through real-time traffic changes; multiple RL agents jointly control traffic signals at all intersections to improve the overall network efficiency. Each agent in the Markov game has its own reward, which improves training efficiency but also brings challenges of mixed games; and the Nash equilibrium (NE) is inconsistent with the global optimality, which limits the performance improvement; equal treatment of intersections is not enough to meet the needs of real ATSC scenarios. In real traffic scenarios, the importance of intersections varies depending on static characteristics such as location and surrounding environment. Existing methods ignore the realistic factors brought about by the change of intersection importance, such as location and environment, resulting in the failure to consider all intersections in a balanced manner when optimizing traffic efficiency. For example, intersections near hospitals or fire stations require higher attention to ensure smooth traffic and respond to time-sensitive emergencies. Ignoring the heterogeneous importance of different intersections may affect the traffic efficiency of special intersections.

[0004] Therefore, a traffic signal control method that considers the importance priority of intersections is needed. Summary of the invention

[0005] In view of this, the present invention provides a traffic signal control method based on multi-agent reinforcement learning and intersection importance, constructs a multi-agent reinforcement learning framework for traffic signal control at intersections with different importance priorities, improves the traffic efficiency of special intersections, and thus improves the overall traffic efficiency.

[0006] To this end, the present invention provides the following technical solutions:

[0007] Traffic signal control method based on multi-agent reinforcement learning and intersection importance, including:

[0008] The CM-MG model is constructed based on intersection priorities to set the decision sequence for all agents in the traffic grid;

[0009] Select the best action for each agent in the decision sequence through the GNSD-Light network;

[0010] The GNSD-Light network includes:

[0011] Obtain environmental information through the observation representation layer;

[0012] The historical action features of each agent are extracted through the forward action representation layer, so that the member agents can understand and cooperate with the decisions of the predecessor;

[0013] Output the best action of each agent sequentially through the Q output layer;

[0014] The observation representation layer comprises:

[0015] Extract spatial features through a concise spatial processing module based on relative position encoding;

[0016] Integrate neighborhood observation features through residual graph attention network.

[0017] Furthermore, the CM-MG model is constructed based on intersection priorities to set a decision sequence for all agents in the traffic grid, including:

[0018] The important intersection agents in the traffic grid are called captain nodes, which make decisions based on their own observations;

[0019] The captain node has the highest decision priority and serves as a predecessor node of the member nodes;

[0020] Ordinary intersection agents in the traffic grid are called member nodes, which make judgments and decisions based on their own observation information and the decisions of the corresponding predecessor nodes;

[0021] Set all captain nodes as the initial node set for the search, and obtain the decision sequence through the BFS algorithm.

[0022] Furthermore, the spatial features are extracted by a concise spatial processing module based on relative position coding:

[0023] The raw observation data of the intersection is subjected to feature extraction through a multi-layer perceptron to generate input embeddings;

[0024] The input embedding is combined with the dimension d in Position embedding Add together to get the observation embedding for the residual graph attention network:

[0025]

[0026] Among them, W and b are the weight matrix and bias vector to be learned; position embedding is the vector derived from the relative position encoding of the Transformer.

[0027] Furthermore, the integrating neighborhood observation features through the residual graph attention network includes:

[0028] Enhanced environment perception and multi-agent collaboration through GAT layers and residual networks:

[0029]

[0030] Among them, h (t) is the observation embedding, L is the number of GAT layers, and the input of each layer is the output from the previous layer;

[0031] The residual module combines the output of each GATs:

[0032]

[0033] Furthermore, the front-end operation presentation layer includes:

[0034] The multiple precursor influences on each member are balanced and integrated through the mean field module;

[0035] Temporal information is extracted through a concise temporal processing module based on relative position encoding.

[0036] Furthermore, the balancing and integration of multiple precursor influences on each member through the mean field module includes:

[0037] By taking the average vector Multiply by the forward action vector matrix to combine the various forward action vectors of the agent into a single one-dimensional vector; each element of which is a prime number to ensure unique influences from different directions; forward action vector:

[0038]

[0039] in, is the subject τ i A collection of predecessors.

[0040] Furthermore, the extracting of time information by a concise time processing module based on relative position coding includes:

[0041] Through MLP, the agent τ i The predecessor historical action vector Perform feature extraction to obtain the original action embedding and form an action embedding with time information:

[0042]

[0043] Among them, W and b are the weight matrix and bias vector to be learned, and the position embedding PE is the vector matrix obtained based on the time sequence of historical actions.

[0044] Furthermore, the Q output layer comprises:

[0045] Agent τ from the observation representation module i The observation shows is input to the attention module as a query;

[0046] Predecessor historical action representation from the predecessor action representation module are input to the attention module as keys and values;

[0047] The attention module adaptively assigns weights based on the importance and relevance of historical predecessor actions to optimize decisions.

[0048] Furthermore, the attention module adaptively assigns weights according to the importance and relevance of historical predecessor actions to optimize decision making, including:

[0049] Deploying residual connections around the attention module, the output is:

[0050]

[0051] After the linear layer processing, the Q value distribution of each action is derived:

[0052]

[0053] Among them, W and b are the weight matrix and bias vector to be learned;

[0054] The best action for each agent is selected by using a maximization operation based on the Q-value.

[0055] Advantages and positive effects of the present invention:

[0056] The method of the present invention uses an autoregressive framework to enable the shared network to make sequential decisions for multiple intersections starting from the captain intersection according to the predefined order of CM-MG; GNSD-Light adopts RGATs, combines the graph attention network and the residual network, and introduces concise spatial processing and temporal processing modules to extract spatiotemporal features from intersection observations and forward actions to obtain sufficient state representation; the rationality of CM-MG is qualitatively verified through the classic mixed matrix game. The traffic efficiency of special intersections is improved, thereby improving the traffic efficiency of the overall transportation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 This is a framework diagram of sequential decision-making traffic light control based on graph neural network in an embodiment of the present invention;

[0059] Figure 2 is the index m in the embodiment of the present invention q The learning curve of the reinforcement learning algorithm.

[0060] Figure 3 is the index m in the embodiment of the present invention qc The learning curve of the reinforcement learning algorithm.

[0061] Figure 4 The figure compares the generalization results of GNSD-Light in the embodiment of the present invention with those based on reinforcement learning in synthetic1 and synthetic2. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] In various social cooperation scenarios, taking a football team as an example, the captain is the core with the strongest personal or command ability, and other team members form a hierarchical structure with decreasing decision priority according to the distance from the captain (or role positioning). By making the overall policy more inclined to the captain, teamwork can be more easily achieved. Inspired by this, the present invention provides a traffic signal control method based on multi-agent reinforcement learning and intersection importance; the ATSC problem with different intersection priorities is modeled as a multi-agent sequential decision model, called the captain-member Markov game (CM-MG); in CM-MG, all agents follow a specific decision sequence τ, which is determined by the BFS algorithm (breadth-first search) starting from multiple captain nodes, that is, multi-starting breadth-first search. The multi-starting breadth-first search first sets all captain nodes as the initial node set for the search, and starts the entire search process through the BFS algorithm. Among these nodes, special intersections are called captain nodes, which have the highest decision priority and can make decisions directly based on their own observations, thereby ensuring the smooth flow of traffic. Ordinary intersections are called member nodes, which make judgments and decisions based on their own observation information and the decisions of the corresponding predecessor nodes. The goal is to optimize the traffic efficiency of the local and overall road networks respectively. The cooperation requirements of CM-MG are realized through the principle of sequential decision-making. The team leader chooses actions first, and the members make decisions in sequence according to the order of width priority in the road network after hearing the intention of the predecessor. CM-MG effectively guarantees the efficiency of overall traffic and special intersections through this priority-based asynchronous decision-making mechanism. Through the collaborative MARL algorithm, the Sequence Decision Deep Graph Network (GNSD-Light), it is used to learn the optimal joint strategy that meets the learning objectives of the team leader and members at the same time. In order not to be limited by the size of the intersection, GNSDLight adopts an autoregressive framework, and all agents make decisions in sequence according to a predetermined order by sharing the same decision model. In addition, in order to obtain sufficient state representation, a GNSD-Light spatiotemporal representation module based on relative position coding is designed according to the characteristics of the ATSC scene. And through sufficient experimental verification and theoretical analysis, it is shown that the method of the present invention effectively balances the traffic efficiency of the entire road network and special intersections.

[0065] Combination Figure 1 Framework diagram, further illustrating the sequential decision traffic light control network (GNSD-Light) based on graph neural network of the present invention: The network includes three modules: observation representation layer, forward action representation layer and Q output layer. The observation representation layer and forward action representation layer input the observation representation and forward action representation to the Q output layer respectively. The Q output layer analyzes these representations and selects the best operation for each agent in the sequence.

[0066] S1 observation representation layer:

[0067] In order to obtain as comprehensive environmental information as possible, GNSD-Light is used as the observation representation layer; including:

[0068] S11, through a concise spatial processing module (SP) based on relative position encoding, fully extract spatial features;

[0069] S12. Residual Graph Attention Networks (RGATs) are used to integrate neighborhood observation features, supplement local observation information, help capture invisible but critical states, and enhance the agent's comprehensive perception of the environment.

[0070] The specific process includes:

[0071] SP first performs preliminary feature extraction on the original observation data of the intersection through a multi-layer perceptron (MLP) to generate input embeddings. Then, the input embeddings are combined with the dimension d in The location of the embedded PE τ Add together to get the observation embedding for RGATs:

[0072]

[0073] Among them, W and b are the weight matrix and bias vector to be learned; position embedding is the vector derived from the relative position encoding of the Transformer.

[0074] RGATs enhances environmental perception and multi-agent collaboration by stacking GAT layers and adding residual networks. The residual connections around the GAT layers improve stability. The format of GATs is as follows:

[0075]

[0076] Among them, h (t) is the observation embedding with spatial information processed by SP for all intersections, L is the number of GAT layers, and the input of each layer is the output from the previous layer.

[0077] Finally, the residual module in RGATs combines the output of each GATs as follows:

[0078]

[0079] S2 forward drive as presentation layer

[0080] The forward action representation layer extracts the historical action features of each agent, enabling member agents to better understand and collaborate with the forward decision, including:

[0081] S21. Given that each member may be affected by multiple predecessors, these effects are balanced and integrated through the mean field (MF) module;

[0082] S22: In order to make full use of the historical action information of the predecessor, the time information is extracted through a concise time processing module (TP) based on relative position coding.

[0083] The specific process includes:

[0084] MF combines the various forward action vectors of the agent into a single one-dimensional vector by averaging the vector This is done by multiplying the previous driving action vector matrix; each element in it is a prime number to ensure unique influence from different directions (up, down, left, right).

[0085] MF is formatted as And the forward drive vector:

[0086]

[0087] in, is the subject τ i A collection of predecessors.

[0088] In this embodiment, Set to {3, 5, 7, 11}, in order to consider the delay effect on the local intersection due to long roads or few vehicles, the predecessor historical actions of the previous k steps are considered To more accurately reflect the impact of the front drive intersection.

[0089] Taking into full consideration the temporal nature of the action, the predecessor historical action vector with temporal features is extracted through TP. TP firstly performs MLP on the agent τ i The predecessor historical action vector Perform preliminary feature extraction to obtain the original action embedding and form an action embedding with time information:

[0090]

[0091] Among them, W and b are the weight matrix and bias vector to be learned, and the position embedding PE is the vector matrix obtained based on the time sequence of historical actions.

[0092] S3Q value output layer

[0093] The Q-value output layer sequentially outputs the best actions for each agent, including:

[0094] S31, Agent τ from the observation representation module i The observation shows is input to the attention module as a query;

[0095] S32, the predecessor historical action representation from the predecessor action representation module (The captain’s forward action is replaced by the zero vector of zero action) is input to the attention module as key and value;

[0096] S33. The attention module adaptively assigns weights based on the importance and relevance of historical forward actions to optimize decisions.

[0097] The specific process includes:

[0098] Similar to RGATs, the Q-value output layer also deploys residual connections around the attention module, and the output is shown as:

[0099]

[0100] Go through a linear layer to derive the Q-value distribution for each action:

[0101]

[0102] Among them, W and b are the weight matrix and bias vector to be learned;

[0103] The best action for each agent is selected by using a maximization operation based on the Q-value.

[0104] The following is a specific experiment to further illustrate the technical effect of the method of the present invention:

[0105] The experiments were conducted on the Cityflow simulator, using traffic data from Hangzhou, Jinan, and two synthetic traffic datasets. These datasets include traffic flow information for each road network, such as the number of vehicles, speed, route, etc. Considering the low original traffic volume in Hangzhou and Jinan, the traffic flow data on the real road network was moderately expanded to fully verify the performance of our algorithm under high traffic and congestion conditions. The datasets for the four scenarios are shown in Table 1:

[0106] Table 1

[0107]

[0108] Experiment 1: Comparing GNSD-Light with MARL methods

[0109] Figure 1The learning curves of the MARL method in four traffic scenarios are shown. Each curve is the average result obtained by repeating five times with different random seeds. GNSD-Light outperforms other methods, especially on the Jinan network, where the high traffic flow and density changes increase the complexity of training, thus highlighting the advantages of the algorithm. GNSD-Light performs well in both overall and special intersection performance, thanks to its priority-based decision-making and enhanced cooperation around key intersections.

[0110] Experiment 2: Ablation study

[0111] Six indicators are selected to evaluate the traffic conditions of general intersections and special intersections.

[0112] travel_time: Average travel time for all vehicles:

[0113]

[0114] in, and denote the entry and exit times of vehicle v, respectively, and Represents a vehicle entering the road network.

[0115] queue: Average queue length at each intersection:

[0116]

[0117] queue captain: Average queue length at special intersections (captain):

[0118]

[0119] Throughput: The total number of vehicles arriving at the destination:

[0120]

[0121] in, is a group of vehicles that reach their destination within 3600 seconds.

[0122] delay: Average delay rate at each intersection:

[0123]

[0124] training_time: total training time (hours) for 150 rounds on 3090 GPU, expressed as m t .

[0125] Table 2

[0126]

[0127]

[0128] To evaluate the contribution of each component in GNSD-Light (captain-member architecture, mean field, spatiotemporal feature extraction), ablation studies were performed on four networks. GNSD-Light (-CM) does not rely on predecessor decisions, but directly computes the Q value from the observation representation through a multi-layer perceptron (MLP) in the Q output layer. GNSD-Light (-MF) takes the mean vector of MF as Simplified to {1, 1, 1, 1}, GNSD-Light (-TC) and (-SC) omit temporal feature extraction (TP) and spatial feature extraction (SP&GAT), respectively.

[0129] The results of the performance comparison in Table 2 confirm the importance of these components and show the significant effect of the captain-member structure in improving the efficiency of the road network, especially at special intersections. The adjustment improves performance by better identifying the preceding actions. The optimization of temporal and spatial feature extraction enhances decision support by accurately reflecting the details of the environment.

[0130] Experiment 3: Promotion experiment

[0131] In reality, traffic is often subject to fluctuations. For example, rainy and snowy weather can cause the city speed limit to drop from 40km / h to 20km / h, which in turn affects traffic flow. To address this variability and the lack of real traffic data under such conditions in open source datasets, the generalization performance of the algorithm in the face of traffic changes was verified in a synthetic network. Two sets of generalization experiments were conducted: one set compared GNSD-Light with other MARL methods, and the other set compared GNSD-Light with ablation methods.

[0132] Figure 2 It shows that GNSD-Light has better generalization ability than other MARL methods in the variation of datasets.

[0133] The experimental results show that the GNSD-Light method effectively improves the traffic efficiency of special intersections and optimizes the overall traffic efficiency.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic signal control method based on multi-agent reinforcement learning and intersection importance, characterized in that: include: The CM-MG model is constructed based on intersection priorities to set the decision sequence for all agents in the traffic grid; Select the best action for each agent in the decision sequence through the GNSD-Light network; The GNSD-Light network includes: Obtain environmental information through the observation representation layer; The historical action features of each agent are extracted through the forward action representation layer, so that the member agents can understand and cooperate with the decisions of the predecessor; The best action of each agent is output sequentially through the Q-value output layer; The observation representation layer comprises: Extract spatial features through a concise spatial processing module based on relative position encoding; Integrate neighborhood observation features through residual graph attention network.

2. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The CM-MG model is constructed based on intersection priorities to set decision sequences for all agents in the traffic grid, including: The important intersection agents in the traffic grid are called captain nodes, which make decisions based on their own observations; The captain node has the highest decision priority and serves as a predecessor node of the member nodes; Ordinary intersection agents in the traffic grid are called member nodes, which make judgments and decisions based on their own observation information and the decisions of the corresponding predecessor nodes; Set all captain nodes as the initial node set for the search, and obtain the decision sequence through the BFS algorithm.

3. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The spatial features are extracted by a concise spatial processing module based on relative position coding: The raw observation data of the intersection is subjected to feature extraction through a multi-layer perceptron to generate input embeddings; The input embedding is combined with the dimension d in Position embedding Add together to get the observation embedding for the residual graph attention network: Among them, W and b are the weight matrix and bias vector to be learned; position embedding is the vector derived from the relative position encoding of the Transformer.

4. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The integration of neighborhood observation features through the residual graph attention network includes: Enhanced environment perception and multi-agent collaboration through GAT layers and residual networks: … Among them, h( t ) is the observation embedding, L is the number of GAT layers, and the input of each layer is the output from the previous layer; The residual module combines the output of each GATs:

5. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The front-end operation presentation layer includes: The multiple precursor influences on each member are balanced and integrated through the mean field module; Temporal information is extracted through a concise temporal processing module based on relative position encoding.

6. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The mean field module balances and integrates multiple precursor influences on each member, including: By taking the average vector Multiply the forward action vector matrix to combine the various forward action vectors of the agent into a single one-dimensional vector; each element of which is a prime number to ensure unique influences from different directions; the forward action vector: in, is the subject τ i A collection of predecessors.

7. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The extracting of time information by a concise time processing module based on relative position coding includes: Through MLP, the agent τ i The predecessor historical action vector Perform feature extraction to obtain the original action embedding and form an action embedding with time information: Among them, W and b are the weight matrix and bias vector to be learned, and the position embedding PE is the vector matrix obtained based on the time sequence of historical actions.

8. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 1 is characterized in that: The Q value output layer comprises: Agent τ from the observation representation module i The observation shows is input to the attention module as a query; Predecessor historical action representation from the predecessor action representation module are input to the attention module as keys and values; The attention module adaptively assigns weights based on the importance and relevance of historical predecessor actions to optimize decisions.

9. The traffic signal control method based on multi-agent reinforcement learning and intersection importance according to claim 8, characterized in that: The attention module adaptively assigns weights based on the importance and relevance of historical predecessor actions to optimize decision making, including: Deploying residual connections around the attention module, the output is: After the linear layer processing, the Q value distribution of each action is derived: Among them, W and b are the weight matrix and bias vector to be learned; The best action for each agent is selected by using a maximization operation based on the Q-value.

Images