Traffic signal cooperative control method based on multi-agent reinforcement learning

Abstract

The invention relates to the technical field of traffic control, in particular to a traffic signal cooperative control method based on multi-agent reinforcement learning, and the method comprises the following steps: modeling each signal lamp intersection in a road network as an agent, and constructing a distributed multi-agent control environment; dividing the whole road network into a plurality of small subnets according to spatial correlation, and sharing and aggregating traffic information in the subnets through a neighborhood information sharing mechanism; utilizing a space-time diagram attention network to extract traffic state characteristics including a space-time dependency relationship in an intersection agent and a neighborhood thereof; and defining a traffic state, a traffic signal control strategy, a reward and punishment function, a network architecture and a target function required by training of the distributed intelligent agent, and carrying out joint training on the distributed intelligent agent until a training target is achieved. According to the invention, the real-time sensing capability of the signal control intelligent agent to the traffic flow dynamic state can be improved, and the collaborative decision-making capability among multiple intelligent agents is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and more particularly to a traffic signal cooperative control method based on multi-agent reinforcement learning. Background Art

[0002] Traffic congestion has become one of the core problems restricting the sustainable development of urban economy and ecological environment. Due to limited land resources, it is difficult for urban traffic infrastructure to relieve pressure through continuous large-scale expansion. Therefore, intelligent traffic signal control has become one of the key technical paths to improve traffic efficiency. By reasonably allocating road rights and phase passing times on the original road infrastructure, traffic congestion can be alleviated. With the continuous development of big data and artificial intelligence, traffic signal control (TSC) strategies based on deep reinforcement learning (DRL) have become a feasible method to improve traffic efficiency and reduce carbon emissions.

[0003] The existing technical solutions have the following defects:

[0004] 1) Lack of an end-to-end traffic signal cooperative control method. In the above technical solutions, each module including state extraction, graph structure modeling, and Q-value calculation is independent of each other, and it is difficult to establish an efficient mapping relationship between traffic states and signal control strategies.

[0005] 2) The ability to extract spatio-temporal characteristics of traffic flow at signal intersections is limited, which restricts the actual effect of traffic signal control. Usually, only graph neural networks are relied on to model the traffic state in the spatial dimension, and it is difficult to effectively extract the volatility and periodicity characteristics of traffic flow in the time dimension, thereby limiting the in-depth exploration of the spatio-temporal evolution law of multi-intersection traffic flow.

[0006] 3) The cooperation mechanism between traffic signal control agents is not clear. In the existing technical solutions, multiple signal intersections are modeled as a joint agent, and the computational complexity will increase with the increase in the number of signal agents, making it difficult to solve the problem of multi-intersection control in large-scale road networks. There are also technical solutions that use independent agents to control traffic signals, ignoring the complex interaction effects between different agent decisions. In short, the existing technical solutions lack a systematic modeling of the neighborhood definition, information sharing, and cooperation strategy of the multi-agent system, resulting in insufficient information interaction between agents and making it difficult to achieve global optimal control.

[0007] 4) The traffic signal control decision lacks a safety constraint modeling mechanism. Existing traffic signal control strategies usually directly learn the action probability distribution or adopt a greedy strategy for phase switching, lacking explicit modeling of key time safety constraints, which easily leads to frequent switching or overly long holding of green lights, thereby causing a decline in traffic efficiency and potential safety hazards. In addition, the existing technical solutions that use constraint resolution methods to penalize unsafe strategies cannot be effectively implemented. Because if unsafe signal timing occurs during the actual operation of traffic signals, it will cause irreparable losses to life and property. Summary of the Invention

[0008] In view of this, the present invention provides a traffic signal collaborative control method based on multi-agent reinforcement learning, which can improve the real-time perception ability of signal control agents for traffic flow dynamics, enhance the collaborative decision-making ability among multi-agents, and achieve safety constraints and stable execution of traffic signal control behaviors.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A traffic signal collaborative control method based on multi-agent reinforcement learning, comprising the following steps:

[0011] Model each signalized intersection in the road network as an agent to construct a distributed multi-agent control environment;

[0012] Divide the entire road network into several small sub-networks according to spatial correlation, and share and aggregate traffic information within the sub-networks through a neighborhood information sharing mechanism;

[0013] Use a spatio-temporal graph attention network to dynamically process the aggregated traffic information and extract traffic state features containing spatio-temporal dependence relationships in the intersection agent and its neighborhood;

[0014] Define the traffic state, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of the distributed agents, and conduct joint training on the distributed agents until the training objective is achieved;

[0015] The trained distributed agents perform collaborative control of traffic signals based on the real-time traffic state features of each intersection in the road network.

[0016] Further, the step of dividing the entire road network into several small sub-networks according to spatial correlation and realizing the sharing and aggregation of traffic information within the sub-networks through a neighborhood information sharing mechanism includes:

[0017] Calculate the spatial distance between every two signal intersections. When the distance between two adjacent intersections satisfies they are classified as a neighborhood pair, where represents the spatial distance threshold;

[0018] Construct a neighborhood-based connection structure-edge index matrix E i , defined as follows:

[0019] ;

[0020] where each column represents the edge connection from the i-th agent to the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, (i, π i ) represents the connection of the i-th agent to the π i -th adjacent agent, (π i , i) represents the reverse connection of the i-th agent to the π i -th adjacent agent, and π i represents the number of agents in the neighborhood of agent i, excluding itself;

[0021] Establish an information exchange channel between neighborhood pairs to share real-time key traffic information; at each control step, the local signal control intersection shares its traffic state and reward with adjacent signal control intersections, and at the same time receives feedback from the traffic environment.

[0022] Furthermore, the agent obtains shared rewards from itself and multiple agents within its neighborhood. The reward sharing mechanism is as follows:

[0023] ;

[0024] where and are the rewards of the j-th adjacent agent and the i-th agent at the t-th control step, respectively; represents the shared reward obtained by the agent.

[0025] Furthermore, the spatio-temporal graph attention network is used to dynamically process the aggregated traffic information to extract traffic state features including spatio-temporal dependencies in the intersection agent and its neighborhood, including:

[0026] Construct a time multi-head self-attention module based on the long short-term memory network and the multi-head attention mechanism. Input the aggregated original traffic information into the LSTM layer to obtain the time dynamic features of each intersection; introduce the output of the LSTM into the multi-head attention mechanism, identify and strengthen the attention to key historical traffic states by calculating the correlation between different control steps in parallel; after multi-head attention aggregation of all nodes within the neighborhood range over multiple control steps, through residual connection and normalization processing, obtain the dynamic traffic features of the signal intersection at different time scales;

[0027] Construct a spatial multi-head self-attention module based on a dynamic graph attention network to model the spatial dependence relationship between intersections. Take the traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix as inputs, and introduce the graph attention mechanism to capture the spatial interaction relationship between intersections. For the target agent i, the dynamic graph attention network calculates the multi-head attention weights for its adjacent agents respectively, and splices the outputs of each head to form the comprehensive feature representation of the i-th agent at the current spatial layer. After performing dynamic graph attention processing on all nodes within the neighborhood range for multiple control steps, the final traffic state features containing spatio-temporal dependence relationships are obtained.

[0028] Furthermore, the traffic state information of the i-th agent in the distributed agents at control step t includes the average queue length and average waiting time of the vehicles in the inbound lanes controlled by each phase. The state is defined as follows:

[0029] ;

[0030] where P represents the number of signal phases of the agent, and are the average waiting time and average queue length of the j-th agent in the neighborhood at control step t for the p-th green light phase respectively;

[0031] Considering the evolution law of the traffic state within multiple control time steps, the state representation of the i-th agent at control step t is modified to:

[0032] ;

[0033] where L represents the length of the historical time window.

[0034] Furthermore, the definition process of the traffic signal control strategy of the distributed agents includes:

[0035] Regarding the minimum and maximum green light times as safety constraint conditions in reinforcement learning, after considering the safety constraint conditions, the action representation of the i-th agent at control step t is:

[0036] ;

[0037] where, represents the duration of the p-th green light phase of the i-th agent at time t, and respectively represent the preset minimum and maximum green light times of the $i$-th agent; 0 means to maintain the current green light phase if the duration of the current green light phase has not reached the minimum green light time; 1 means to switch to the next green light phase if the duration of the current green light phase reaches or exceeds the maximum green light time; if the current phase duration is between the minimum and maximum green light times, the agent adaptively determines whether to maintain the current phase or switch to the next phase.

[0038] Furthermore, the reward and punishment function of the distributed agent consists of three sub-goals: waiting time, carbon emissions, and intersection pressure. The local reward function of the $i$-th agent at control step $t$ is defined as:

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] where represents the weight coefficients of the three sub-goals, represents the set of approach lanes of the $i$-th agent, represents the total vehicle waiting time on the $h$-th approach lane of the $i$-th agent at control step $t$, represents the total vehicle carbon emissions on the $h$-th approach lane of the $i$-th agent at control step $t$, represents the pressure of the $i$-th agent at control step $t$; represents the waiting time of the -th vehicle on the $h$-th approach lane under the control of the $i$-th agent, represents the total number of vehicles on the $h$-th approach lane under the control of the $i$-th agent; represents approach lanes and exit lanes, represents the set of lanes under the control of the $i$-th agent, represents the absolute value of the difference between the number of vehicles entering and leaving the intersection; represents the -th vehicle on the $h$-th lane under the control of the $i$-th agent's fuel consumption rate per unit time, and represent the vehicle speed and acceleration; represents the fuel carbon emission coefficient.

[0044] Furthermore, each agent adopts an Actor-Critic architecture, including a policy network and an old policy network and the value network , where the policy network and the old policy network belong to the Actor network, and the value network belongs to the Critic network; the Actor network is used to approximate the policy function and is responsible for dynamically generating control policies according to the collected traffic state characteristics, and the Critic network uses the feedback rewards of the traffic environment to evaluate the control performance of the Actor network;

[0045] Both the Actor network and the Critic network integrate spatio-temporal attention neural networks. Each network includes an input layer, an intermediate layer, and an output layer. Among them, the intermediate layer is a fully connected neural network containing a spatio-temporal graph attention network with a non-linear activation function, a residual layer, and a normalization layer.

[0046] Furthermore, the objective function for optimizing the parameters of the i-th agent's Actor network is expressed as:

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] Among them, represents the clipping rate, represents the generalized advantage estimation parameter, represents the discount factor, represents the weight coefficient of entropy; represents the expected average value of the objective function in a small batch of training samples; clip represents the clipping function, which is used to clip the upper and lower limits of the target value; represents the generalized advantage function of the i-th agent at the control step t 's importance sampling weight, which is used to calculate the probability ratio of the new and old policy networks selecting actions in the state, quantifying the difference between the new and old policies; and respectively represent the new policy and the old policy network of the Actor network in the i-th agent at the control step t state selecting action 's probability; represents the policy entropy, which is used to enhance the exploration ability of the agent and prevent premature convergence to the local optimum; and respectively represent the value estimates in and states, denotes the shared reward of the \(i\)-th agent at control step \(t\). denotes the TD target value of the \(i\)-th agent at control step \(t\), where \(D\) represents the maximum number of steps for experience collection.

[0052] Furthermore, the objective function for optimizing the parameters of the \(i\)-th agent's Critic network is expressed as:

[0053] ;

[0054] where denotes the value function of the \(i\)-th agent at control step \(t\), denotes the weight coefficient.

[0055] As can be seen from the above technical solutions, compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) By introducing a neighborhood-aware information sharing and cooperation mechanism, the present invention realizes the interaction of state and reward information among agents in both the distributed training and control stages, strengthening the cooperation relationship among TSC agents. This mechanism not only avoids the high-dimensional calculation problem caused by centralized modeling in the existing centralized multi-agent architecture but also overcomes the problems of isolated decision-making and insufficient cooperation ability of each agent in the independent multi-agent architecture, effectively improving the coordinated control ability of multiple intersections in the road network and enhancing the overall traffic control performance.

[0057] (2) The present invention proposes a spatio-temporal attention network structure that integrates the time and space dimensions, breaking through the limitation of existing methods that only model traffic features in a single dimension. It can accurately capture the volatility and periodicity characteristics of traffic flow over time and identify the complex traffic correlation structure between intersections, thereby enhancing the expressive and decision-making abilities of traffic state modeling.

[0058] (3) The present invention constructs an end-to-end traffic signal adaptive cooperative control architecture, realizing the full-process mapping from the original traffic state observation to the output of the signal control strategy. Different from the serial structure in the prior art where state extraction, graph modeling, and decision optimization are separated from each other. By integrating perception, modeling, and decision-making in a unified deep reinforcement learning framework, the present invention can output the optimal signal control strategy in real time in a dynamic traffic environment.

[0059] (4) The present invention proposes a reinforcement learning method for safety constraints. By introducing explicit signal timing constraints such as maximum and minimum green light times, it effectively guides agents to learn strategies that comply with traffic safety rules during the training stage and dynamically adjusts the signal phase switching behavior during the execution stage, achieving double guarantees for the stability and safety of the signal control decision-making process. Description of the Drawings

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0061] Figure 1 It is a flowchart of the traffic signal cooperative control method based on multi-agent reinforcement learning provided by the present invention;

[0062] Figure 2 It is a schematic diagram of the distributed multi-agent control environment provided by the present invention;

[0063] Figure 3 It is a schematic diagram of the structure of the temporal multi-head self-attention module provided by the present invention;

[0064] Figure 4 It is a schematic diagram of the structure of the spatial multi-head self-attention module provided by the present invention;

[0065] Figure 5 It is a comparison diagram of the control effects of the method of the present invention and the existing method in terms of queue length;

[0066] Figure 6 It is a comparison diagram of the control effects of the method of the present invention and the existing method in terms of waiting time;

[0067] Figure 7 It is a comparison diagram of the control effects of the method of the present invention and the existing method in terms of carbon emissions;

[0068] Figure 8 It is a schematic diagram of the distribution of the agent neighborhood range of the present invention;

[0069] Figures 9(a) - 9(d) It is a schematic diagram of the difference in the time ranges concerned by different attention heads of the agent under the temporal multi-head self-attention mechanism provided by the present invention;

[0070] Figures 10(a) - 10(d) In the spatial multi-head self-attention mechanism provided by the present invention, the agent The attention distribution to neighboring agents. Detailed implementation manners

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0072] As Figure 1 shown, an embodiment of the present invention discloses a traffic signal collaborative control method based on multi-agent reinforcement learning, which is characterized by including the following steps:

[0073] S1. Model each signalized intersection in the road network as an agent, and construct a distributed multi-agent control environment;

[0074] S2. Divide the entire road network into several small subnets according to spatial correlation, and share and aggregate traffic information within the subnets through a neighborhood information sharing mechanism;

[0075] S3. Dynamically process the aggregated traffic information using a spatio-temporal graph attention network, and extract traffic state features containing spatio-temporal dependence relationships in the intersection agent and its neighborhood;

[0076] S4. Define the traffic state, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of the distributed agents, and jointly train the distributed agents until the training objective is reached;

[0077] S5. The trained distributed agents perform collaborative control of traffic signals based on the real-time traffic state features of each intersection in the road network.

[0078] Next, the above steps will be described in detail.

[0079] S1. As Figure 2 shown on the right, the regional road network consists of I signalized intersections, where each signalized intersection is modeled as an agent based on the Actor-Critic architecture, thus constructing a distributed multi-agent control environment.

[0080] S2. The step of dividing the entire road network into several small subnets according to spatial correlation and realizing the sharing and aggregation of traffic information within the subnets through a neighborhood information sharing mechanism specifically includes:

[0081] As Figure 2 shown on the left, calculate the spatial distance between two signal intersections. When the distance between two adjacent intersections satisfies , these two intersections are classified as a neighborhood pair, where represents the spatial distance threshold;

[0082] Construct a neighborhood-based connection structure, and let a more efficient edge index matrix E i replace the traditional adjacency matrix A (if intersection i is connected to adjacent agent j, then ), and the edge index matrix E i , which is defined as follows:

[0083] (1);

[0084] Among them, each column represents the edge connection between the i-th agent and the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, (i, π i ) represents the connection between the i-th agent and the π i -th adjacent agent, (π i , i) represents the reverse connection between the i-th agent and the π i -th adjacent agent, and π i represents the number of agents in the neighborhood of agent i, excluding itself;

[0085] As Figure 2 shown by the dash-dotted arrow in the figure, an information exchange channel is established between neighborhood pairs to share key traffic information in real time; at each control step, the local signal control intersection shares its traffic state and rewards with adjacent signal control intersections, while receiving feedback from the traffic environment. To encourage the collaborative decision-making of multiple TSC agents within the neighborhood range, at each control step, the agent obtains a shared reward from itself and multiple agents within the neighborhood range, and the reward sharing mechanism is as follows:

[0086] (2);

[0087] Among them, and are the rewards of the j-th adjacent agent and the i-th agent at the t-th control step, respectively; represents the shared reward obtained by the agent.

[0088] S3. The above-mentioned use of the spatio-temporal graph attention network to dynamically process the aggregated traffic information, extract the traffic state features including spatio-temporal dependence relationships in the intersection agent and its neighborhood, specifically includes:

[0089] S31. Construct a temporal multi-head self-attention module based on the long short-term memory network (LSTM) and the multi-head attention mechanism to extract the dynamic traffic features of the signal intersection at different time scales.

[0090] Specifically, as Figure 3As shown in the figure, the aggregated original traffic information is input into the LSTM layer to obtain the time dynamic features of each intersection; the output of the LSTM is introduced into the multi-head attention mechanism, and the correlation between different control steps is calculated in parallel to identify and strengthen the attention to key historical traffic states; after multi-head attention aggregation of all nodes within the neighborhood range in multiple control steps, through residual connection and normalization processing, the dynamic traffic features of the signal intersection at different time scales are obtained.

[0091] In the \(i\)-th TSC agent, the output of the time multi-head self-attention mechanism in the \(l\)-th layer (multi-head attention layer) of the time multi-head self-attention module is , and its calculation formula is:

[0092] (3);

[0093] (4);

[0094] (5);

[0095] Among them, represents the projection matrices of query, key, and value, represents the projection matrix of the aggregated multi-head attention value, \(M\) represents the number of attention heads, represents the dimension of the key and . represents the attention weight of the \(i\)-th TSC agent in the \(m\)-th attention head of the \(l\)-th layer, and respectively represent the feature vectors of the \(i\)-th TSC agent in the \((l - 1)\)-th and \(l\)-th layers. Here, the feature vector has been processed by the previous LSTM network for time modeling, and contains the dynamic information of the historical traffic state of this intersection. In addition, to alleviate the problem of gradient disappearance in the deep architecture, this module adopts residual connection and layer normalization operations. To ensure the smooth progress of the residual connection, the input and output must have the same dimension. Therefore, all intermediate layer neural networks (including the LSTM layer and the multi-head attention layer) in the time multi-head self-attention module share the same dimension . After multi-head attention aggregation of all nodes within the neighborhood range in multiple time steps, the final output of the \(l\)-th layer can be obtained.

[0096] S32. As Figure 4As shown in the figure, a spatial multi-head self-attention module based on the Graph Attention Technology v2 (GATv2) is constructed to model the spatial dependence relationships between intersections. Specifically, the traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix are taken as inputs together, and the graph attention mechanism is introduced to capture the spatial interaction relationships between intersections; for the target TSC agent i, the dynamic graph attention network calculates the multi-head attention weights for its adjacent agents respectively, and concatenates the outputs of each head to form the comprehensive feature representation of the i-th agent at the current spatial layer. This feature integrates key traffic information from adjacent intersections, such as vehicle waiting time and queue length, thus enhancing the model's ability to perceive the operation trend of local traffic flow.

[0097] The attention score between the adjacent i-th and j-th TSC agents and the normalized attention score as well as the output of the spatial multi-head attention mechanism of the i-th agent at the l-th layer and the t-th time step in the spatial attention module are calculated by the following formulas respectively:

[0098] (6);

[0099] (7);

[0100] (8);

[0101] where, is the number of adjacent agents of the i-th TSC agent, is the learnable attention weight vector, are the feature transformation matrices from the source node to the target node within the small neighborhood range respectively; LeakyReLU is an activation function used to improve the non-linear modeling ability; represents the attention score of the TSC agent i to the adjacent agent j in the l-th layer and the m-th attention head; represents the attention weight of the TSC agent i to the adjacent agent j in the l-th layer and the m-th attention head;

[0102] Similarly, the spatial multi-head self-attention module adopts residual connection and layer normalization to overcome the problem of vanishing gradients that appears as the number of network layers increases.

[0103] After performing dynamic graph attention processing on all nodes within the neighborhood range for multiple control steps, the final output of the l-th layer can be obtained, which serves as the traffic state feature containing spatio-temporal dependence relationships.

[0104] S4. Define the traffic state, traffic signal control strategy, reward and punishment function, network architecture, and objective function required for training of the distributed agents, and conduct joint training on the distributed agents until the training objective is achieved.

[0105] S41. Define the traffic state of the distributed Actor-Critic agents.

[0106] The traffic state information of the i-th agent at control step t includes the average queue length and average waiting time of the vehicles in the incoming lanes controlled by each phase. The state is defined as follows:

[0107] (9);

[0108] where P represents the number of signal phases of the agent, and are respectively the average waiting time and average queue length of the p-th green light phase of the j-th agent in the neighborhood at control step t. To unify the dimensions, normalize and clip each index.

[0109] Since only contains the traffic states of adjacent intersections within the current time step, the decision-making of the TSC agent also needs to consider the evolution law of traffic states over multiple time steps. Therefore, represent the state of the i-th agent at control step t as modified to:

[0110] (10);

[0111] where L represents the length of the historical time window.

[0112] S42. Define the traffic signal control strategy of the distributed Actor-Critic agents.

[0113] Under the assumption of a fixed phase sequence, the available actions are defined as maintaining the current green light phase or switching to the next green light phase. However, frequent switching of the green light phase or too long green light time will have a negative impact on the traffic efficiency and safety of signalized intersections. To solve this problem, the present invention regards the minimum and maximum green light times as safety constraint conditions in reinforcement learning, guides policy learning of safe behaviors during the training phase, and ensures constraint satisfaction through action enforcement rules during the execution phase. After considering the safety constraint conditions, the action representation of the i-th agent at control step t is:

[0114] (11);

[0115] where denotes the duration of the \(p\)-th green light phase for the \(i\)-th agent at time \(t\). and respectively denote the preset minimum and maximum green light times for the \(i\)-th agent; 0 indicates that if the current green light phase duration has not reached the minimum green light time, the current green light phase is maintained; 1 indicates that if the current green light phase duration reaches or exceeds the maximum green light time, the next green light phase is switched; if the current phase duration is between the minimum and maximum green light times, the agent will adaptively determine whether to maintain the current phase or switch to the next phase based on the historical traffic features extracted by the spatio-temporal attention network and the status information shared from adjacent intersections, so as to achieve more efficient local response and regional collaborative control. This mechanism realizes the integration of reinforcement learning and safety control strategies by shielding actions that do not meet the time constraints, ensuring that traffic control agents always meet the basic safety specifications of traffic signal control during the learning process.

[0116] S43. Define the reward and punishment function of the distributed Actor-Critic agent.

[0117] The goal of the present invention is to improve the road network traffic efficiency under signalized intersection control and reduce carbon emissions. Therefore, the reward function of the TSC agent consists of three sub-goals: waiting time, carbon emissions, and intersection pressure. Based on the neighborhood information sharing mechanism, as shown in Equation (2), the shared reward function of agents within the neighborhood consists of the local reward functions of adjacent intersections. The local reward function of the \(i\)-th agent at control step \(t\) is defined as:

[0118] (12);

[0119] (13);

[0120] (14);

[0121] (15);

[0122] where denotes the weight coefficients of the three sub-goals, denotes the set of approach lanes of the \(i\)-th agent, denotes the total vehicle waiting time on the \(h\)-th approach lane of the \(i\)-th agent at control step \(t\), denotes the total vehicle carbon emissions on the \(h\)-th approach lane of the \(i\)-th agent at control step \(t\), denotes the pressure of the \(i\)-th agent at control step \(t\); denotes the waiting time of the -th vehicle on the \(h\)-th approach lane under the control of the \(i\)-th agent represents the total number of vehicles on the h-th import lane controlled by the i-th agent; represents import lanes and export lanes, represents the set of lanes controlled by the i-th agent, represents the absolute value of the difference in the number of vehicles entering and leaving the intersection; represents the fuel consumption rate of the vehicle on the h-th lane controlled by the i-th agent per unit time, and represents the speed and acceleration of the vehicle; represents the fuel carbon emission coefficient, and the CO2 emission coefficient of gasoline vehicles is 2.31 kg / L.

[0123] S44. Define the deep neural network of the distributed Actor-Critic agent.

[0124] Each agent adopts the Actor-Critic architecture, including a policy network , an old policy network and a value network . Among them, the policy network and the old policy network belong to the Actor network, and the value network belongs to the Critic network; the Actor network is used to approximate the policy function and is responsible for dynamically generating control policies according to the collected traffic state characteristics, and the Critic network uses the feedback reward of the traffic environment to evaluate the control performance of the Actor network;

[0125] Both the Actor network and the Critic network integrate spatio-temporal attention neural networks to improve the cooperative decision-making performance by extracting the spatial topology and traffic flow dynamics between adjacent signal agents. Each network includes an input layer, an intermediate layer, and an output layer. Among them, the intermediate layer is a fully connected neural network containing a spatio-temporal graph attention network with a non-linear activation function, a residual layer, and a normalization layer.

[0126] Among them, the dimension of the input layer is 64 and the ReLU activation function is adopted. The dimensions of the time and space modules in the spatio-temporal attention layer . The dimension of the output layer of the Actor network is 2, and the dimension of the output layer of the Critic network is 1. The Actor network outputs probabilities through the Softmax activation function and samples to obtain actions, while the Critic network outputs the value estimation of the control effect of the distributed Actor network through the ReLU activation function.

[0127] S45. Define the optimization objective function of the parameters of the distributed Actor-Critic agent model.

[0128] The objective function for optimizing the parameters of the Actor network of the i-th agent is expressed as:

[0129] (16);

[0130] (17);

[0131] (18);

[0132] (19);

[0133] where represents the shear rate, represents the Generalized Advantage Estimation (GAE) parameter, represents the discount factor, represents the weight coefficient of entropy; represents the expected average value of the objective function in a mini-batch of training samples; clip represents the clipping function for clipping the target value to upper and lower bounds; represents the generalized advantage function of the i-th agent at control step t the importance sampling weight of for calculating the probability ratio of the new and old policy networks selecting actions in the and respectively represent the new policy and the old policy network of the Actor network in the i-th agent at control step t when selecting an action in the represents the policy entropy, which is used to enhance the exploration ability of the agent and prevent premature convergence to the local optimum; and respectively represent the value estimates in and states, represents the shared reward of the i-th agent at control step t, represents the TD target value of the i-th agent at control step t, and D represents the maximum number of steps for experience collection.

[0134] The objective function for optimizing the parameters of the Critic network of the i-th agent is expressed as:

[0135] (20);

[0136] where represents the target value function of the i-th agent at control step t, represents the weight coefficient.

[0137] S46. Reinforce the learning of distributed multi - agents based on the spatio - temporal attention network and the neighborhood information sharing mechanism. The specific training process includes:

[0138] 1) Set the total number of training rounds , with the number of simulation control steps T = 180 per round, the number of mini - batch data samples B = 256, and the spatial distance threshold .

[0139] 2) Use the method of orthogonal initialization, that is, initialize the weight matrix and bias in the network as an orthogonal matrix and a constant 0 respectively, to obtain the Actor network parameters and Critic network parameters of the initialized distributed traffic TSC agent. At the same time, initialize the experience replay pool .

[0140] 3) Divide the road network into several small neighborhoods according to the edge index matrix .

[0141] 4) The distributed agent uses formulas (9) and (10) to collect the initial traffic flow state from itself and adjacent agents within the neighborhood range .

[0142] 5) At each control time step t, the distributed agent obtains the traffic state information from the intersections within the neighborhood and sends it into the Actor network; the old policy network of the Actor generates the control policy through the spatio - temporal graph attention network. If , then execute within seconds; otherwise, execute the yellow light within seconds, and then execute within seconds.

[0143] 6) Concatenate the control policies generated by multiple Actor networks into a global traffic signal control action , which acts on the regional road network. The agent obtains the shared reward and the traffic flow state of the next control time step of each agent according to formulas (12) - (15) and formula (2), and stores in the experience replay pool of each agent.

[0144] 7) Repeat steps 5) - 6) until the number of training data reaches the size of the experience replay pool.

[0145] 8) Retrieve the state data stored in the distributed agent experience pool and , which are used as the inputs for each distributed agent's Critic network, and obtain the state values and respectively. Calculate the generalized advantage function using formulas (18)-(19).

[0146] 9) Each agent samples B mini-batch data samples from its respective experience pool for training.

[0147] 10) Calculate the objective function using formula (20) and update the parameters of each agent's Critic network using the Adam gradient descent algorithm . Calculate the objective function using formulas (16)-(17) and update each agent's Actor network using the Adam gradient descent algorithm.

[0148] 11) Update the old policy network parameters with the new policy network parameters of the Actor network , and clear the agent's experience replay pool.

[0149] 12) Repeat steps 4)-11) until the total number of training episodes is reached, and save the trained model parameters.

[0150] S5. The trained distributed agents perform cooperative control of traffic signals based on the real-time traffic state characteristics of each intersection in the road network.

[0151] To evaluate the performance of the control method of the present invention, independent deep Q-network learning (IQL), independent advantage actor-critic algorithm (IA2C), independent proximal policy optimization algorithm (IPPO), fixed-timed control (FTC), and max-pressure control (MP) are used as comparison objects. A simulation experiment is carried out on a traffic road network taking the Beijing Yizhuang area as an example. This road network has 36 intersections, and each intersection has 4 phases. After the training process of the traffic signal cooperative control SANIS-MARL model proposed by the present invention converges stably, the control effect is evaluated by statistically analyzing the queue length, waiting time, and carbon emissions of vehicles under different control methods. The comparison of the control effects is as followsFigures 5 - 7 As shown, the solid line represents the control effect of the present invention. Since the method proposed in the present invention integrates the spatiotemporal graph attention module and the neighborhood information sharing mechanism, they jointly promote more effective coordination and decision-making between adjacent intersections, so the performance in reducing vehicle queue length, waiting time, carbon emissions, etc. is better than other control methods.

[0152] At the same time, the multi-head attention score matrix in the SANIS-MARL model is visualized as a heat map, which provides a way to explain the impact of spatiotemporal graph attention based on neighborhood information sharing on cooperative multi-junction TSC. Figure 8 Represents a target TSC agent and multiple adjacent TSC agents The neighborhood range. Figures 9(a) - 9(d) These neighboring TSC agents are shown in Figure 8 A matrix of spatiotemporal attention scores over multiple time steps for the neighborhood shown.

[0153] like Figures 9(a) - 9(d) As shown in the figure, in the time multi-head attention mechanism of the traffic signal control (TSC) agent, the time ranges that different attention heads focus on are significantly different. Among them, the second and third attention heads mainly focus on the recent traffic status, emphasizing the direct impact of short-term dynamic characteristics on signal control decisions; while the first and fourth attention heads tend to focus on earlier historical time steps, reflecting the important role of long-term traffic trends in strategy optimization. Obviously, this multi-scale modeling in the time dimension enhances the model's ability to represent the evolution of traffic status.

[0154] Figures 10(a) - 10(d) Demonstrated the TSC agent in the spatial multi-head attention mechanism The distribution of attention to neighboring agents. Neighbors that are close or closely connected receive higher attention weights, while intersections that are not directly related in space have weights close to zero. This result shows that dividing neighborhoods based on road network distance and extracting effective spatial features can help improve the collaborative control effect among multiple agents. Overall, the visualization results of spatiotemporal graph attention verify its effectiveness in enhancing the perception ability of signal control agents and optimizing the decision-making process.

[0155] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0156] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A traffic signal collaborative control method based on multi-agent reinforcement learning, characterized in that It includes the following steps: Model each signalized intersection in the road network as an agent to construct a distributed multi-agent control environment; Divide the entire road network into several small sub-networks according to spatial correlation, and share and aggregate traffic information within the sub-networks through a neighborhood information sharing mechanism; Dynamically process the aggregated traffic information using a spatio-temporal graph attention network to extract traffic state features containing spatio-temporal dependencies in intersection agents and their neighborhoods; Define the traffic state, traffic signal control strategy, reward function, network architecture and objective function required for training of distributed agents, and jointly train the distributed agents until the training objective is achieved; The trained distributed agents perform cooperative control of traffic signals based on the real-time traffic state features at each intersection in the road network.

2. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 1, wherein The process of dividing the entire road network into several small sub-networks according to spatial correlation and realizing the sharing and aggregation of traffic information within the sub-networks through a neighborhood information sharing mechanism includes: Calculate the spatial distance between pairwise signal intersections. When the distance between two adjacent intersections satisfies , the two intersections are classified as a neighborhood pair, where represents the spatial distance threshold. Construct a neighborhood-based connection structure-edge index matrix E i , which is defined as follows: ； Among them, each column represents the edge connection between the i-th agent and the j-th adjacent agent, (i, i) represents the self-connection of the i-th agent, (i, π i ) represents the connection between the i-th agent and the π i -th adjacent agent, (π i , i) represents the reverse connection between the i-th agent and the π i -th adjacent agent, and π i represents the number of agents in the neighborhood of agent i, excluding itself; Establish an information exchange channel between neighborhood pairs for real-time sharing of key traffic information; at each control step, the local signal-controlled intersection shares its traffic state and reward with neighboring signal-controlled intersections, and at the same time receives feedback from the traffic environment.

3. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 2, characterized in that, At each control step, the agent obtains shared rewards from itself and multiple agents within its neighborhood. The reward sharing mechanism is as follows: ； wherein, and are the rewards of the j-th agent and the i-th agent adjacent at the t-th control step, respectively; represents the shared reward obtained by the agent.

4. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 1, characterized in that The process of dynamically processing the aggregated traffic information using a spatio-temporal graph attention network to extract traffic state features containing spatio-temporal dependencies in intersection agents and their neighborhoods includes: Construct a time multi-head self-attention module based on a long short-term memory network and a multi-head attention mechanism. Input the aggregated original traffic information into the LSTM layer to obtain the time dynamic features of each intersection; introduce the output of the LSTM into the multi-head attention mechanism, and identify and strengthen the attention to key historical traffic states by calculating the correlation between different control steps in parallel; after performing multi-head attention aggregation on all nodes within the neighborhood range over multiple control steps, obtain the dynamic traffic features of the signal intersection at different time scales through residual connection and normalization processing. Construct a spatial multi-head self-attention module based on a dynamic graph attention network to model the spatial dependence relationships between intersections. Take the traffic features processed by the temporal multi-head attention module and the corresponding edge index matrix as inputs together, and introduce the graph attention mechanism to capture the spatial interaction relationships between intersections. For the target agent i, the dynamic graph attention network calculates the multi-head attention weights for its adjacent agents respectively, and splices the outputs of each head to form the comprehensive feature representation of the i-th agent at the current spatial layer. After performing dynamic graph attention processing on all nodes within the neighborhood range for multiple control steps, the final traffic state features containing spatio-temporal dependence relationships are obtained.

5. The traffic signal collaborative control method based on multi-agent reinforcement learning according to claim 1, characterized in that The traffic state information of the $i$-th agent in the distributed agents at control step $t$ includes the average queue length and average waiting time of the vehicles in the approach lanes controlled by each phase, and the state is defined as follows: ； where P represents the number of signal phases of the agent, and are the average waiting time and average queue length of the j-th agent in the neighborhood at the control step t for the p-th green light phase, respectively; Considering the evolution law of traffic states within multiple control time steps, the state of the $i$-th agent at control time step $t$ is represented as Modified to: ； Among them, L represents the length of the historical time window.

6. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 1, characterized in that The definition process of the traffic signal control strategy of the distributed agent includes: Regarding the minimum and maximum green light times as safety constraints in reinforcement learning, after considering the safety constraints, the action representation of the i-th agent at control step t is as follows: ； Among them, represents the p-th green light phase duration of the i-th agent at time t, and respectively represent the preset minimum and maximum green light times of the i-th agent; 0 means that if the current green light phase duration has not reached the minimum green light time, the current green light phase is maintained; 1 means that if the current green light phase duration reaches or exceeds the maximum green light time, switch to the next green light phase; if the current phase duration is between the minimum green light time and the maximum green light time, the agent adaptively determines whether to maintain the current phase or switch to the next phase.

7. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 1, characterized in that The reward and punishment function of the distributed agent consists of three sub-goals: waiting time, carbon emissions, and intersection pressure. The local reward function of the $i$-th agent at control step $t$ is defined as: ； ； ； ； Among them, represents the weight coefficients of three sub-goals, represents the set of import lanes of the i-th agent, represents the total vehicle waiting time of the i-th agent on the h-th import lane at control step t, represents the total vehicle carbon emissions on the h-th import lane of the i-th agent at control step t, represents the pressure of the i-th agent at control step t; represents the waiting time of the vehicle on the h-th import lane under the control of the i-th agent, represents the total number of vehicles on the h-th import lane under the control of the i-th agent; represents the import lane and the export lane, represents the set of lanes under the control of the i-th agent, represents the absolute value of the difference between the number of vehicles entering and leaving the intersection; represents the vehicle's fuel consumption rate per unit time on the h-th lane under the control of the i-th agent, and represents the vehicle's speed and acceleration; represents the fuel carbon emission coefficient.

8. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 1, wherein Each agent adopts the Actor-Critic architecture, including a policy network , an old policy network and a value network . Among them, the policy network and the old policy network belong to the Actor network, and the value network belongs to the Critic network; the Actor network is used to approximate the policy function and is responsible for dynamically generating control policies according to the collected traffic state characteristics, and the Critic network uses the feedback rewards of the traffic environment to evaluate the control performance of the Actor network; Both the Actor network and the Critic network integrate a spatio-temporal attention neural network. Each network includes an input layer, an intermediate layer and an output layer. Among them, the intermediate layer is a fully connected neural network containing a spatio-temporal graph attention network with a non-linear activation function, a residual layer and a normalization layer.

9. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 8, wherein, The objective function for optimizing the parameters of the Actor network of the i-th agent is expressed as: ； ； ； ； Among them, represents the shear rate, represents the generalized advantage estimation parameter, represents the discount factor, represents the weight coefficient of entropy; represents the expected average value of the objective function in a small batch of training samples; clip represents the clipping function used to clip the target value to upper and lower limits; represents the generalized advantage function of the i-th agent at control step t The importance sampling weight of is used to calculate the probability ratio of the new and old policy networks selecting actions in the state quantifies the difference between the new and old policies; and respectively represent the new policy and the old policy network of the Actor network in the i-th agent at control step t selecting actions in the state probability; represents the policy entropy, which is used to enhance the exploration ability of the agent and prevent premature convergence to the local optimum; and respectively represent the value estimations in and states, represents the shared reward of the i-th agent at control step t, represents the TD target value of the i-th agent at control step t, and D represents the maximum number of steps for experience collection.

10. The traffic signal cooperative control method based on multi-agent reinforcement learning according to claim 9, characterized in that, The objective function for optimizing the parameters of the Critic network of the i-th agent is expressed as: ； Among them, represents the objective value function of the $i$-th agent at control step $t$, represents the weight coefficient.

Images