Traffic signal coordinated control method based on multi-agent reinforcement learning

Abstract

The present invention relates to the field of traffic control technology, and specifically to a method for coordinated traffic signal control based on multi-agent reinforcement learning, comprising the following steps: modeling each traffic light intersection in a road network as an agent to construct a distributed multi-agent control environment; dividing the entire road network into several small subnets based on spatial correlation, and sharing and aggregating traffic information within the subnets through a neighborhood information sharing mechanism; utilizing a spatiotemporal graph attention network to extract traffic state characteristics containing spatiotemporal dependencies between intersection agents and their neighborhoods; defining the traffic state, traffic signal control strategy, reward and punishment functions, network architecture, and objective functions required for training of distributed agents, and jointly training the distributed agents until the training goal is achieved. The present invention can enhance the real-time perception capability of signal control agents of traffic flow dynamics and enhance the collaborative decision-making capabilities among multiple agents.

Description

Technical Field

[0001] The present invention relates to the field of traffic control technology, and more specifically to a traffic signal collaborative control method based on multi-agent reinforcement learning. Background Art

[0002] Traffic congestion has become a core issue hindering the sustainable development of urban economies and ecological environments. Due to limited land resources, urban transportation infrastructure cannot be alleviated through continued large-scale expansion. Therefore, intelligent traffic signal control has become a key technological path to improving traffic efficiency. Traffic congestion can be alleviated by rationally allocating right-of-way and phase travel times within existing road infrastructure. With the continuous development of big data and artificial intelligence, traffic signal control (TSC) strategies based on deep reinforcement learning (DRL) have become a viable method for improving traffic efficiency and reducing carbon emissions.

[0003] The existing technical solutions have the following defects:

[0004] 1) Lack of an end-to-end coordinated traffic signal control approach. The aforementioned technical solutions, including state extraction, graph structure modeling, and Q-value calculation, are independent of each other, making it difficult to establish an efficient mapping between traffic states and signal control strategies.

[0005] 2) The ability to extract spatiotemporal features of traffic flow at signalized intersections is limited, which restricts the actual effectiveness of traffic signal control. Usually, only graph neural networks are used to model traffic states in the spatial dimension, making it difficult to effectively extract the volatility and periodicity of traffic flow in the temporal dimension. This in turn limits the in-depth exploration of the spatiotemporal evolution of traffic flow at multiple intersections.

[0006] 3) The coordination mechanism between traffic signal control agents is unclear. Existing solutions model multiple signalized intersections as a single joint agent. This increases computational complexity as the number of signal agents increases, making it difficult to solve the problem of controlling multiple intersections in large-scale road networks. Other solutions use independent agents to control traffic signals, ignoring the complex interactions between the decisions of different agents. In short, existing solutions lack systematic modeling of neighborhood definition, information sharing, and coordination strategies for multi-agent systems. This leads to insufficient information exchange between agents and makes it difficult to achieve global optimal control.

[0007] 4) Traffic signal control decisions lack a safety constraint modeling mechanism. Existing traffic signal control strategies typically directly learn action probability distributions or employ greedy strategies for phase switching. These strategies lack explicit modeling of critical time safety constraints, which can easily lead to frequent green light switching or excessively long green light holds, resulting in decreased traffic efficiency and potential safety hazards. Furthermore, existing technical solutions employing constraint solutions that penalize unsafe signal timing strategies are not practical. Unsafe signal timing in actual traffic signal operation can cause irreversible loss of life and property. Summary of the Invention

[0008] In view of this, the present invention provides a traffic signal collaborative control method based on multi-agent reinforcement learning, which can improve the real-time perception ability of the signal control agent of traffic flow dynamics, enhance the collaborative decision-making ability among multiple agents, and achieve safe constraints and stable execution of traffic signal control behaviors.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A traffic signal coordinated control method based on multi-agent reinforcement learning includes the following steps:

[0011] Model each traffic light intersection in the road network as an agent and build a distributed multi-agent control environment;

[0012] The entire road network is divided into several small subnetworks based on spatial correlation, and traffic information is shared and aggregated within the subnetwork through a neighborhood information sharing mechanism;

[0013] The spatiotemporal graph attention network is used to dynamically process the aggregated traffic information and extract the traffic state features containing spatiotemporal dependencies between the intersection agent and its neighborhood.

[0014] Define the traffic status, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of distributed intelligent agents, and jointly train the distributed intelligent agents until the training objectives are achieved;

[0015] The trained distributed intelligent agents coordinate the control of traffic signals based on the real-time traffic status characteristics of each intersection in the road network.

[0016] Furthermore, the entire road network is divided into several small sub-networks according to spatial correlation, and traffic information is shared and aggregated within the sub-networks through a neighborhood information sharing mechanism, including:

[0017] Calculate the spatial distance between two signalized intersections. When the distance between two adjacent intersections meets , the two intersections are classified as a neighborhood pair, where represents the spatial distance threshold;

[0018] Constructing a neighborhood-based connection structure - edge index matrix E i , defined as follows:

[0019] ;

[0020] Among them, each column represents the edge connection from the i-th agent to the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, (i, π i ) represents the relationship between the i-th agent and the π-th agent i adjacent agents are connected, (π i , i) represents the relationship between the i-th agent and the π-th agent i Neighboring agents are connected in reverse order, π i represents the number of agents in the neighborhood of agent i, excluding itself;

[0021] An information exchange channel is established between neighboring pairs to share key traffic information in real time. At each control step, the local signalized intersection shares its traffic status and rewards with neighboring signalized intersections while receiving feedback from the traffic environment.

[0022] Furthermore, the agent receives shared rewards from itself and multiple agents in the neighborhood. The reward sharing mechanism is as follows:

[0023] ;

[0024] in, and are the rewards of the adjacent j-th agent and the i-th agent at t control step respectively; Represents the shared reward obtained by the agent.

[0025] Furthermore, the spatiotemporal graph attention network is used to dynamically process the aggregated traffic information and extract traffic state features containing spatiotemporal dependencies between the intersection agent and its neighborhood, including:

[0026] A temporal multi-head self-attention module based on a long short-term memory network and a multi-head attention mechanism is constructed. The aggregated raw traffic information is input into the LSTM layer to obtain the temporal dynamic characteristics of each intersection. The LSTM output is introduced into the multi-head attention mechanism, which identifies and strengthens attention to key historical traffic states by parallel computing the correlations between different control steps. After multi-head attention aggregation of all nodes in the neighborhood within multiple control steps, residual connections and normalization are performed to obtain the dynamic traffic characteristics of signalized intersections at different time scales.

[0027] A spatial multi-head self-attention module based on a dynamic graph attention network is constructed to model the spatial dependency between intersections. The traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix are used as inputs, and a graph attention mechanism is introduced to capture the spatial interaction between intersections. For the target agent i, the dynamic graph attention network performs self-attention on its adjacent Each agent calculates the multi-head attention weights separately and splices the outputs of each head to form the comprehensive feature representation of the 𝑖th agent at the current spatial layer; after performing dynamic graph attention processing for multiple control steps on all nodes in the neighborhood, the final traffic state features containing spatiotemporal dependencies are obtained.

[0028] Furthermore, the traffic state information of the i-th agent in the distributed agent at control step t includes the average queue length and average waiting time of vehicles in the import lane controlled by each phase, and the state The definition is as follows:

[0029] ;

[0030] Where P represents the number of signal phases of the agent, and are the average waiting time and average queue length of the j-th agent in the neighborhood at the p-th green light phase at control step t;

[0031] Considering the evolution of traffic status within multiple control steps, the state of the i-th agent at control step t is represented as Modify to:

[0032] ;

[0033] Where L represents the length of the historical time window.

[0034] Furthermore, the process of defining the traffic signal control strategy of the distributed agent includes:

[0035] The minimum and maximum green light times are regarded as safety constraints in reinforcement learning. After considering the safety constraints, the action of the i-th agent at control step t is expressed as for:

[0036] ;

[0037] in, represents the duration of the pth green light phase of the ith agent at time t, and They represent the minimum and maximum green light times preset for the i-th agent respectively; 0 means that if the duration of the current green light phase has not reached the minimum green light time, the current green light phase is maintained; 1 means that if the duration of the current green light phase reaches or exceeds the maximum green light time, the next green light phase is switched to; if the current phase duration is between the minimum green light time and the maximum green light time, the agent adaptively determines whether to maintain the current phase or switch to the next phase.

[0038] Furthermore, the reward and punishment function of the distributed agent consists of three sub-goals: waiting time, carbon emissions, and intersection pressure. The local reward function of the i-th agent at control step t is Defined as:

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] in, represents the weight coefficients of the three sub-goals, represents the set of import lanes of the i-th agent, represents the total waiting time of vehicles in the hth import lane at the control step t of the i-th agent, represents the total carbon emissions of vehicles on the h-th import lane at control step t of the i-th agent, represents the pressure of the i-th agent at control step t; represents the hth import lane under the control of the i-th agent The waiting time of a car, represents the total number of vehicles on the h-th import lane under the control of the i-th agent; Indicates the import lane and the exit lane, represents the set of lanes controlled by the i-th agent, It represents the absolute value of the difference between the number of vehicles entering and leaving the intersection; represents the hth lane under the control of the i-th agent The fuel consumption rate of the vehicle per unit time, and Indicates the speed and acceleration of the vehicle; Represents the fuel carbon emission coefficient.

[0044] Furthermore, each agent adopts the Actor-Critic architecture, including the policy network , old strategy network and value network Among them, the policy network and the old policy network belong to the Actor network, and the value network belongs to the Critic network; the Actor network is used to approximate the policy function and is responsible for dynamically generating control strategies based on the collected traffic state characteristics. The Critic network uses the feedback rewards of the traffic environment to evaluate the control performance of the Actor network;

[0045] Both the Actor network and the Critic network integrate spatiotemporal attention neural networks. Each network consists of an input layer, an intermediate layer, and an output layer. The intermediate layer is a fully connected neural network with a spatiotemporal graph attention network containing a nonlinear activation function, a residual layer, and a normalization layer.

[0046] Furthermore, the objective function of the network parameter optimization of the i-th agent Actor is expressed as:

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] in, represents the shear rate, represents the generalized advantage estimation parameter, represents the discount factor, Represents the weight coefficient of entropy; Represents the expected average value of the objective function in a small batch of training samples; clip represents the clipping function, which is used to clip the upper and lower limits of the target value; represents the generalized advantage function of the ith agent at control step t The importance sampling weights of are used to calculate the new and old policy networks in Select action in status The probability ratio quantifies the difference between the new and old strategies; and They represent the new strategy and old strategy of the Actor network in the i-th agent at control step t. Select action in status probability; Represents the policy entropy, which is used to enhance the agent's exploration ability and prevent premature convergence to the local optimum; and Respectively expressed in and The value estimation in the state, represents the shared reward of the ith agent at control step t, represents the TD target value of the i-th agent at control step t, and D represents the maximum number of steps of experience collection.

[0052] Furthermore, the objective function of the parameter optimization of the Critic network of the i-th agent is expressed as:

[0053] ;

[0054] in, represents the objective value function of the ith agent at control step t, Represents the weight coefficient.

[0055] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) By introducing a neighborhood-aware information sharing and collaboration mechanism, this paper achieves the interaction of state and reward information between agents during both the distributed training and control phases, thereby strengthening the collaborative relationship between TSC agents. This mechanism not only avoids the high-dimensional computational issues caused by centralized modeling in existing centralized multi-agent architectures, but also overcomes the isolated decision-making and insufficient collaboration capabilities of each agent in independent multi-agent architectures. It effectively enhances the coordinated control capabilities of multiple intersections in a road network and improves overall traffic control performance.

[0057] (2) The present invention proposes a spatiotemporal attention network structure that integrates time and space dimensions, breaking through the limitation of existing methods that only model traffic characteristics in a single dimension. It can accurately capture the volatility and periodicity of traffic flow over time and identify the complex traffic correlation structure between intersections, thereby improving the expression and decision-making capabilities of traffic state modeling.

[0058] (3) The present invention constructs an end-to-end traffic signal adaptive collaborative control architecture, which realizes the full process mapping from raw traffic state observation to signal control strategy output. Different from the serial structure in the existing technology where state extraction, graph modeling and decision optimization are separated from each other, the present invention realizes the integrated fusion of perception, modeling and decision making under a unified deep reinforcement learning framework, which can output the optimal signal control strategy in real time in a dynamic traffic environment.

[0059] (4) This paper proposes a reinforcement learning method for safety constraints. By introducing explicit signal timing constraints such as maximum and minimum green light times, it effectively guides the intelligent agent to learn strategies that comply with traffic safety rules during the training phase, and dynamically adjusts the signal phase switching behavior during the execution phase, thereby achieving dual guarantees for the stability and safety of the signal control decision-making process. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0061] Figure 1 Flowchart of the traffic signal collaborative control method based on multi-agent reinforcement learning provided by the present invention;

[0062] Figure 2 A schematic diagram of a distributed multi-agent control environment provided by the present invention;

[0063] Figure 3 A schematic diagram of the structure of the temporal multi-head self-attention module provided by the present invention;

[0064] Figure 4 A schematic diagram of the structure of the spatial multi-head self-attention module provided by the present invention;

[0065] Figure 5 A comparison chart of the control effects of the method of the present invention and the existing method on queue length;

[0066] Figure 6 A comparison chart of the control effects of the method of the present invention and the existing method on waiting time;

[0067] Figure 7 This is a comparison chart of the control effects of the method of the present invention and the existing method on carbon emissions;

[0068] Figure 8 This is a schematic diagram of the neighborhood distribution of intelligent agents in the present invention;

[0069] Figure 9(a)-Figure 9(d) A schematic diagram showing the differences in the time ranges focused on by different attention heads of the agent under the temporal multi-head self-attention mechanism provided by the present invention;

[0070] Figure 10(a)-Figure 10(d) In the spatial multi-head self-attention mechanism provided by the present invention, the agent Attention distribution towards neighboring agents. DETAILED DESCRIPTION

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0072] like Figure 1 As shown, an embodiment of the present invention discloses a traffic signal collaborative control method based on multi-agent reinforcement learning, which is characterized by comprising the following steps:

[0073] S1. Model each traffic light intersection in the road network as an agent and build a distributed multi-agent control environment;

[0074] S2. Divide the entire road network into several small subnetworks based on spatial correlation, and share and aggregate traffic information within the subnetworks through a neighborhood information sharing mechanism;

[0075] S3. Use the spatiotemporal graph attention network to dynamically process the aggregated traffic information and extract the traffic state features containing spatiotemporal dependencies between the intersection agent and its neighborhood.

[0076] S4. Define the traffic status, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of the distributed intelligent agents, and jointly train the distributed intelligent agents until the training goal is achieved;

[0077] S5. The trained distributed intelligent agents coordinate the control of traffic signals based on the real-time traffic status characteristics of each intersection in the road network.

[0078] Below, each of the above steps is described in detail.

[0079] S1, such as Figure 2 As shown on the right, the regional road network consists of I signalized intersections, each of which is modeled as an agent based on the Actor-Critic architecture, thus constructing a distributed multi-agent control environment.

[0080] S2. Dividing the entire road network into several small subnetworks based on spatial correlation and sharing and aggregating traffic information within the subnetworks through a neighborhood information sharing mechanism, specifically including:

[0081] like Figure 2 As shown on the left, the spatial distance between two signalized intersections is calculated. When the distance between two adjacent intersections meets , the two intersections are classified as a neighborhood pair, where represents the spatial distance threshold;

[0082] Construct a neighborhood-based connection structure to make the edge index matrix E more efficient i Instead of the traditional adjacency matrix A (if intersection i is connected to adjacent agent j, then ), the edge index matrix E i , defined as follows:

[0083] (1);

[0084] Among them, each column represents the edge connection from the i-th agent to the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, (i, π i ) represents the relationship between the i-th agent and the π-th agent i adjacent agents are connected, (π i , i) represents the relationship between the i-th agent and the π-th agent i Neighboring agents are connected in reverse order, π i represents the number of agents in the neighborhood of agent i, excluding itself;

[0085] like Figure 2 As shown by the dotted arrows, an information exchange channel is established between neighborhood pairs to share key traffic information in real time. At each control step, the local signalized intersection shares its traffic status and rewards with neighboring signalized intersections, while also receiving feedback from the traffic environment. To encourage collaborative decision-making among multiple TSC agents within a neighborhood, at each control step, an agent receives shared rewards from itself and multiple agents within the neighborhood. The reward sharing mechanism is as follows:

[0086] (2);

[0087] in, and are the rewards of the adjacent j-th agent and the i-th agent at t control step respectively; Represents the shared reward obtained by the agent.

[0088] S3. The method of using the spatiotemporal graph attention network to dynamically process the aggregated traffic information and extract traffic state features containing spatiotemporal dependencies between the intersection agent and its neighborhood includes:

[0089] S31. Construct a temporal multi-head self-attention module based on the long short-term memory (LSTM) network and the multi-head attention mechanism to extract dynamic traffic characteristics of signalized intersections at different time scales.

[0090] Specifically, such as Figure 3As shown in the figure, the aggregated raw traffic information is input into the LSTM layer to obtain the temporal dynamic characteristics of each intersection; the output of the LSTM is introduced into the multi-head attention mechanism, and the correlation between different control steps is calculated in parallel to identify and strengthen the focus on key historical traffic states; after multi-head attention aggregation of all nodes in the neighborhood within multiple control steps, residual connection and normalization are performed to obtain the dynamic traffic characteristics of signalized intersections at different time scales.

[0091] Assume that in the i-th TSC agent, the temporal multi-head attention mechanism output of the l-th layer (multi-head attention layer) in the temporal multi-head self-attention module is , and its calculation formula is:

[0092] (3);

[0093] (4);

[0094] (5);

[0095] in, projection matrices representing queries, keys, and values, Represents the projection matrix of the aggregated multi-head attention value, M represents the number of attention heads, represents the dimension of the key and . represents the attention weight of the 𝑖th TSC agent in the 𝑚th attention head of the lth layer, and Represent the feature vectors of the i-th TSC agent in the l-1th and lth layers respectively. Here, the feature vector The temporal modeling process has been performed by a previous LSTM network, which contains the dynamic information of the historical traffic status of the intersection. In addition, to alleviate the gradient vanishing problem in the deep architecture, the module uses residual connections and layer normalization operations. To ensure that the residual connection can be carried out smoothly, the input and output must have the same dimension. Therefore, all the intermediate neural networks in the temporal multi-head attention module (including the LSTM layer and the multi-head attention layer) share the same dimension. After multi-head attention aggregation of all nodes in the neighborhood within multiple time steps, the final output of the lth layer can be obtained .

[0096] S32, such as Figure 4As shown in the figure, a spatial multi-head self-attention module based on the dynamic graph attention technology v2 (GATv2) is constructed to model the spatial dependency between intersections. Specifically, the traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix are used as input, and the graph attention mechanism is introduced to capture the spatial interaction relationship between intersections; for the target TSC agent i, the dynamic graph attention network Each agent calculates the multi-head attention weights separately and splices the outputs of each head to form a comprehensive feature representation of the 𝑖th agent at the current spatial layer. This feature integrates key traffic information from adjacent intersections, such as vehicle waiting time and queue length, thereby enhancing the model's ability to perceive the local traffic flow operation trend.

[0097] Attention score between adjacent i-th and j-th TSC agents and normalized attention scores And the spatial multi-head attention mechanism output of the i-th agent at the t-th time step of the l-th layer of the spatial attention module The calculation formulas are as follows:

[0098] (6);

[0099] (7);

[0100] (8);

[0101] in, is the number of neighbors of the ith TSC agent, is a learnable attention weight vector, are the feature transformation matrices from the source node to the target node within a small neighborhood; LeakyReLU is an activation function used to improve nonlinear modeling capabilities; represents the attention score of TSC agent i to neighboring agent j in the lth layer and the mth attention head; represents the attention weight of TSC agent i to neighboring agent j in the lth layer and the mth attention head;

[0102] Similarly, the spatial multi-head self-attention module adopts residual connections and layer normalization to overcome the gradient vanishing problem that occurs as the network layers increase.

[0103] After performing dynamic graph attention processing for multiple control steps on all nodes in the neighborhood, the final output of the lth layer can be obtained , as a traffic status feature containing spatiotemporal dependencies.

[0104] S4. Define the traffic status, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of the distributed intelligent agent, and jointly train the distributed intelligent agent until the training goal is achieved.

[0105] S41. Define the traffic state of the distributed Actor-Critic agent.

[0106] The traffic state information of the i-th agent at control step t includes the average queue length and average waiting time of vehicles in the import lane controlled by each phase. The definition is as follows:

[0107] (9);

[0108] Where P represents the number of signal phases of the agent, and are the average waiting time and average queue length of the j-th agent in the neighborhood at the p-th green light phase at control step t, respectively. To unify the dimensions, each indicator is normalized and clipped.

[0109] because It only contains the traffic status of adjacent intersections within the current time step. The decision of the TSC agent also needs to consider the evolution of traffic status within multiple time steps. Therefore, the state of the i-th agent at control step t is represented as Modify to:

[0110] (10);

[0111] Where L represents the length of the historical time window.

[0112] S42. Define the traffic signal control strategy of distributed Actor-Critic agents.

[0113] Under the assumption that the phase sequence is fixed, the available actions are defined as maintaining the current green light phase or switching to the next green light phase. However, frequent switching of green light phases or excessively long green light times will have a negative impact on traffic efficiency and safety at signalized intersections. To address this issue, the present invention considers the minimum and maximum green light times as safety constraints in reinforcement learning, guides the strategy to learn safe behaviors during the training phase, and ensures constraint satisfaction through action enforcement rules during the execution phase. After considering the safety constraints, the action representation of the i-th agent at control step t is for:

[0114] (11);

[0115] in, represents the duration of the pth green light phase of the ith agent at time t, and where represents the minimum and maximum green light times preset for the i-th agent, respectively. 0 means maintaining the current green light phase if it has not yet reached the minimum green light time. 1 means switching to the next green light phase if the current green light phase reaches or exceeds the maximum green light time. If the current phase duration is between the minimum and maximum green light times, the agent will adaptively determine whether to maintain the current phase or switch to the next phase based on historical traffic characteristics extracted through the spatiotemporal attention network and state information shared from adjacent intersections, thereby achieving more efficient local response and regional coordinated control. This mechanism achieves the integration of reinforcement learning and safety control strategies by screening actions that do not meet time constraints, ensuring that the traffic control agent always meets the basic safety standards of traffic signal control during the learning process.

[0116] S43. Define the reward and punishment function of the distributed Actor-Critic agent.

[0117] The goal of this invention is to improve the traffic efficiency of the road network under signalized intersection control and reduce carbon emissions. Therefore, the reward function of the TSC agent is composed of three sub-goals: waiting time, carbon emissions, and intersection pressure. Based on the neighborhood information sharing mechanism, as shown in formula (2), the shared reward function of the agents in the neighborhood is composed of the local reward functions of the adjacent intersections. The local reward function of the i-th agent at control step t is Defined as:

[0118] (12);

[0119] (13);

[0120] (14);

[0121] (15);

[0122] in, represents the weight coefficients of the three sub-goals, represents the set of import lanes of the i-th agent, represents the total waiting time of vehicles in the hth import lane at the control step t of the i-th agent, represents the total carbon emissions of vehicles on the h-th import lane at control step t of the i-th agent, represents the pressure of the i-th agent at control step t; represents the hth import lane under the control of the i-th agent The waiting time of a car, represents the total number of vehicles on the h-th import lane under the control of the i-th agent; Indicates the import lane and the exit lane, represents the set of lanes controlled by the i-th agent, It represents the absolute value of the difference between the number of vehicles entering and leaving the intersection; represents the hth lane under the control of the i-th agent The fuel consumption rate of the vehicle per unit time, and Indicates the speed and acceleration of the vehicle; It represents the carbon emission coefficient of fuel. The CO2 emission coefficient of gasoline vehicles is 2.31kg / L.

[0123] S44. Define a deep neural network for distributed Actor-Critic agents.

[0124] Each agent uses an Actor-Critic architecture, including a policy network , old strategy network and value network Among them, the policy network and the old policy network belong to the Actor network, and the value network belongs to the Critic network; the Actor network is used to approximate the policy function and is responsible for dynamically generating control strategies based on the collected traffic state characteristics. The Critic network uses the feedback rewards of the traffic environment to evaluate the control performance of the Actor network;

[0125] Both the actor and critic networks integrate spatiotemporal attention neural networks, which improve cooperative decision-making performance by extracting the spatial topology and traffic flow dynamics between adjacent signal agents. Each network consists of an input layer, an intermediate layer, and an output layer. The intermediate layer is a fully connected neural network consisting of a spatiotemporal graph attention network with a nonlinear activation function, a residual layer, and a normalization layer.

[0126] Among them, the input layer dimension is 64 and uses ReLU activation function, and the time and space module dimensions in the spatiotemporal attention layer are , the output layer dimension of the Actor network is 2, and the output layer dimension of the Critic network is 1. The Actor network outputs probabilities and samples to obtain actions through the Softmax activation function, while the Critic network outputs a value estimate of the control effect of the distributed Actor network through the ReLU activation function.

[0127] S45. Define the optimization objective function of the distributed Actor-Critic agent model parameters.

[0128] The objective function of the i-th agent Actor network parameter optimization is expressed as:

[0129] (16);

[0130] (17);

[0131] (18);

[0132] (19);

[0133] in, represents the shear rate, represents the generalized advantage estimation (GAE) parameter, represents the discount factor, Represents the weight coefficient of entropy; Represents the expected average value of the objective function in a small batch of training samples; clip represents the clipping function, which is used to clip the upper and lower limits of the target value; represents the generalized advantage function of the ith agent at control step t The importance sampling weights of are used to calculate the new and old policy networks in Select action in status The probability ratio quantifies the difference between the new and old strategies; and They represent the new strategy and old strategy of the Actor network in the i-th agent at control step t. Select action in status probability; Represents the policy entropy, which is used to enhance the agent's exploration ability and prevent premature convergence to the local optimum; and Respectively expressed in and The value estimation in the state, represents the shared reward of the ith agent at control step t, represents the TD target value of the i-th agent at control step t, and D represents the maximum number of steps of experience collection.

[0134] The objective function of the parameter optimization of the Critic network of the i-th agent is expressed as:

[0135] (20);

[0136] in, represents the objective value function of the ith agent at control step t, Represents the weight coefficient.

[0137] S46. Reinforcement learning of distributed multi-agents based on spatiotemporal attention network and neighborhood information sharing mechanism. The specific training process includes:

[0138] 1) Set the total number of training rounds , the number of simulation control steps in each round is T=180, the number of small batch data samples is B=256 and the spatial distance threshold is .

[0139] 2) Use the orthogonal initialization method to initialize the weight matrix and bias in the network to an orthogonal matrix and a constant 0 respectively, and obtain the Actor network parameters of the initialized distributed traffic TSC agent and Critic network parameters At the same time, initialize the experience replay pool .

[0140] 3) According to the edge index matrix Divide the road network into several small neighborhoods.

[0141] 4) The distributed agent uses formulas (9) and (10) to collect the initial traffic flow state from itself and the neighboring agents in the neighborhood. .

[0142] 5) At each control time step t, the distributed agent will obtain traffic status information from the intersections in the neighborhood Feed into the Actor network; Actor's old policy network Generating Control Strategies via Spatiotemporal Graph Attention Networks .if , then in Execute within seconds Otherwise, in Execute the yellow light within seconds, then Execute within seconds .

[0143] 6) Combine the control strategies generated by multiple Actor networks into a global traffic signal control action , acting on the regional road network, the agent obtains shared rewards from itself and the neighboring TSC agents in the neighborhood according to formulas (12)-(15) and formula (2) And the traffic flow state of each agent at the next control time step , and Stored in each agent's experience replay pool.

[0144] 7) Repeat steps 5)-6) until the amount of training data reaches the size of the experience replay pool.

[0145] 8) Retrieve the state data stored in the distributed agent experience pool and , as the input of each distributed agent Critic network, and obtain the state value respectively and , use formulas (18)-(19) to calculate the generalized advantage function .

[0146] 9) Each agent will sample B mini-batches of data from its own experience pool for training.

[0147] 10) Calculate the objective function using formula (20) And Adam gradient descent algorithm to update the critic network parameters of each agent , use formula (16)-(17) to calculate the objective function And Adam gradient descent algorithm to update the Actor network of each agent .

[0148] 11) New strategy network parameters using Actor network Update old policy network parameters , and clear the agent's experience replay pool.

[0149] 12) Repeat steps 4) to 11) until the total number of training rounds is reached , save the trained model parameters.

[0150] S5. The trained distributed intelligent agents coordinate the control of traffic signals based on the real-time traffic status characteristics of each intersection in the road network.

[0151] In order to evaluate the performance of the control method of the present invention, independent deep Q-learning (IQL), independent advantage actor-critic algorithm (IA2C), independent proximal policy optimization algorithm (IPPO), fixed-timed control (FTC) and maximum pressure control (MP) were used as comparison objects, and simulation experiments were carried out on the traffic network in Yizhuang area of ​​Beijing as an example. The network has 36 intersections, and each intersection has 4 phases. After the training process of the traffic signal collaborative control SANIS-MARL model proposed in the present invention has stabilized and converged, the control effect is evaluated by statistically analyzing the queue length, waiting time and carbon emissions of vehicles under different control methods. The control effect comparison is as follows: Figure 5-Figure 7 As shown in the figure, the solid line represents the control effect of the present invention. Since the method proposed in the present invention integrates the spatiotemporal graph attention module and the neighborhood information sharing mechanism, they jointly promote more effective coordination and decision-making between adjacent intersections, so the performance in reducing vehicle queue length, waiting time, carbon emissions, etc. is better than other control methods.

[0152] At the same time, the multi-head attention score matrix in the SANIS-MARL model is visualized as a heat map, which provides a method to explain the impact of spatiotemporal graph attention based on neighborhood information sharing on cooperative multi-junction TSC. Figure 8 Represents a target TSC agent and multiple adjacent TSC agents The neighborhood range of the composition. Figure 9(a)-Figure 9(d) It shows that these neighboring TSC agents Figure 8 Spatiotemporal attention score matrix for multiple time steps within the shown neighborhood.

[0153] like Figure 9(a)-Figure 9(d) As shown in the figure, the time ranges focused on by different attention heads in the Traffic Signal Control (TSC) agent's temporal multi-head attention mechanism vary significantly. The second and third attention heads primarily focus on recent traffic conditions, emphasizing the direct impact of short-term dynamic features on signal control decisions; whereas the first and fourth attention heads tend to focus on earlier historical time steps, reflecting the importance of long-term traffic trends in policy optimization. This multi-scale modeling across time clearly enhances the model's ability to characterize the evolving patterns of traffic conditions.

[0154] Figure 10(a)-Figure 10(d) Demonstrates the TSC agent in the spatial multi-head attention mechanism The distribution of attention towards neighboring agents. Closer or more connected neighbors receive higher attention weights, while intersections with no direct spatial connection receive weights approaching zero. This result demonstrates that dividing neighborhoods based on road network distance and extracting effective spatial features can help improve collaborative control among multiple agents. Overall, the visualization of spatiotemporal graph attention demonstrates its effectiveness in enhancing the perception capabilities of signal control agents and optimizing their decision-making.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0156] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A traffic signal collaborative control method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Model each traffic light intersection in the road network as an agent and build a distributed multi-agent control environment; The entire road network is divided into several small subnetworks based on spatial correlation, and traffic information is shared and aggregated within the subnetwork through a neighborhood information sharing mechanism; The spatiotemporal graph attention network is used to dynamically process the aggregated traffic information and extract the traffic state features containing spatiotemporal dependencies between the intersection agent and its neighborhood. Define the traffic status, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of distributed intelligent agents, and jointly train the distributed intelligent agents until the training objectives are achieved; The trained distributed intelligent agents coordinate traffic signal control based on the real-time traffic status characteristics of each intersection in the road network; The entire road network is divided into several small subnetworks based on spatial correlation, and traffic information is shared and aggregated within the subnetworks through a neighborhood information sharing mechanism, including: Calculate the spatial distance between two signalized intersections. When the distance between two adjacent intersections satisfies d ij When ≤ξ, the two intersections are classified as a neighborhood pair, where ξ represents the spatial distance threshold; Construct the neighborhood-based connection structure - edge index matrix E, which is defined as follows: Each column (i, j) represents the edge connection from the i-th agent to the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, and (i, π i ) represents the relationship between the i-th agent and the π-th agent i Neighboring agents are forward connected, (π i , i) represents the relationship between the i-th agent and the π-th agent i Neighboring agents are connected backward, π i represents the number of agents in the neighborhood of agent i, excluding itself; Establish information exchange channels between neighboring pairs to share key traffic information in real time. At each control step, the local signalized intersection shares its traffic status and rewards with neighboring signalized intersections while receiving feedback from the traffic environment. The spatiotemporal graph attention network is used to dynamically process aggregated traffic information and extract traffic state features containing spatiotemporal dependencies between intersection agents and their neighborhoods, including: Construct a temporal multi-head self-attention module based on long short-term memory networks and multi-head attention mechanisms to extract dynamic traffic characteristics of signalized intersections at different time scales; A spatial multi-head self-attention module based on a dynamic graph attention network is constructed to model the spatial dependency between intersections. The traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix are used as input, and a graph attention mechanism is introduced to capture the spatial interaction relationship between intersections. For the target agent i, the dynamic graph attention network is used to model the spatial dependency between its adjacent π i Each agent calculates the multi-head attention weights separately and splices the outputs of each head to form the comprehensive feature representation of the i-th agent at the current spatial layer; after performing dynamic graph attention processing on all nodes within the neighborhood for multiple control steps, the final traffic state features containing spatiotemporal dependencies are obtained.

2. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: At each control step, the agent receives a shared reward from itself and multiple agents in the neighborhood. The reward sharing mechanism is as follows: Among them, r j,t and r i,t are the rewards of the adjacent j-th agent and the i-th agent at t control step respectively; Represents the shared reward obtained by the agent.

3. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The process of extracting dynamic traffic characteristics of signalized intersections at different time scales includes: A temporal multi-head self-attention module based on a long short-term memory network and a multi-head attention mechanism is constructed. The aggregated raw traffic information is input into the LSTM layer to obtain the temporal dynamic characteristics of each intersection. The output of the LSTM is introduced into the multi-head attention mechanism. By parallel computing the correlation between different control steps, attention to key historical traffic states is identified and strengthened. After multi-head attention aggregation of all nodes in the neighborhood within multiple control steps, the dynamic traffic characteristics of signalized intersections at different time scales are obtained through residual connection and normalization.

4. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The traffic state information of the i-th agent in the distributed agent at control step t includes the average queue length and average waiting time of vehicles in the import lane controlled by each phase, and the state s i,t The definition is as follows: Where P represents the number of signal phases of the agent, and are the average waiting time and average queue length of the j-th agent in the neighborhood at the p-th green light phase at control step t; Considering the evolution of traffic status within multiple control steps, the state of the i-th agent at control step t is represented by s i,t Modify to: Where L represents the length of the historical time window.

5. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The process of defining the traffic signal control strategy of the distributed agent includes: The minimum and maximum green light times are regarded as safety constraints in reinforcement learning. After considering the safety constraints, the action of the i-th agent at control step t is represented by a i,t for: in, represents the duration of the pth green light phase of the ith agent at time t, and They represent the minimum and maximum green light times preset for the i-th agent respectively; 0 means that if the duration of the current green light phase has not reached the minimum green light time, the current green light phase is maintained; 1 means that if the duration of the current green light phase reaches or exceeds the maximum green light time, the next green light phase is switched to; if the current phase duration is between the minimum green light time and the maximum green light time, the agent adaptively determines whether to maintain the current phase or switch to the next phase.

6. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The reward and punishment function of the distributed agent consists of three sub-goals: waiting time, carbon emissions, and intersection pressure. The local reward function r of the i-th agent at control step t is i,t Defined as: Among them, μ, ν, κ represent the weight coefficients of the three sub-goals, L i,in represents the set of import lanes of the i-th agent, represents the total waiting time of vehicles in the hth import lane at the control step t of the i-th agent, represents the total carbon emissions of vehicles on the hth import lane at control step t of the i-th agent, Π i,t represents the pressure of the i-th agent at control step t; represents the hth import lane under the control of the i-th agent The waiting time of a car, represents the total number of vehicles on the hth import lane under the control of the i-th agent; l in ,l out Indicates the import lane and the exit lane, L i represents the set of lanes controlled by the i-th agent, |y(l in )-y(l out )| represents the absolute value of the difference between the number of vehicles entering and leaving the intersection; represents the hth lane under the control of the i-th agent The fuel consumption rate of the vehicle per unit time, v t and a t Indicates the speed and acceleration of the vehicle; C f Represents the fuel carbon emission coefficient.

7. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: Each agent uses an Actor-Critic architecture, including a policy network Old Policy Network and value network The policy network and the old policy network belong to the actor network, while the value network belongs to the critic network. The actor network is used to approximate the policy function and is responsible for dynamically generating control strategies based on collected traffic state characteristics. The critic network uses feedback rewards from the traffic environment to evaluate the control performance of the actor network. Both the Actor network and the Critic network integrate spatiotemporal attention neural networks. Each network consists of an input layer, an intermediate layer, and an output layer. The intermediate layer is a fully connected neural network with a spatiotemporal graph attention network containing a nonlinear activation function, a residual layer, and a normalization layer.

8. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 7 is characterized in that: The objective function of the i-th agent Actor network parameter optimization is expressed as: Where ε represents the shear rate, λ represents the generalized advantage estimation parameter, γ represents the discount factor, and c1 represents the entropy weight coefficient; Represents the expected average value of the objective function in a small batch of training samples; clip represents the clipping function, which is used to clip the upper and lower limits of the target value; represents the generalized advantage function of the ith agent at control step t The importance sampling weights of are used to calculate the new and old policy networks in Select action a in state i,t The probability ratio quantifies the difference between the new and old strategies; and They represent the new strategy and old strategy of the Actor network in the i-th agent at control step t. Select action a in state i,t probability; Represents the policy entropy, which is used to enhance the agent's exploration ability and prevent premature convergence to the local optimum; and Respectively expressed in and The value estimation in the state, represents the shared reward of the ith agent at control step t, δ i,t represents the TD target value of the i-th agent at control step t, and D represents the maximum number of steps of experience collection.

9. The traffic signal coordinated control method based on multi-agent reinforcement learning according to claim 8 is characterized in that: The objective function of the parameter optimization of the Critic network of the i-th agent is expressed as: in, represents the objective value function of the i-th agent at control step t, and c2 represents the weight coefficient.

Images