Multi-agent traffic signal control method based on spatio-temporal convolution attention mechanism

Abstract

The application relates to the technical field of intelligent traffic control, in particular to a multi-agent traffic signal control method based on a space-time convolution attention mechanism, a multi-intersection traffic network of a region is modeled as a multi-agent system, in the learning strategy process, each multi-agent simultaneously considers the influence of the actions of adjacent multi-agents at adjacent time points, so that the multiple multi-agents can cooperatively control the signal lights of the multi-intersections, and a tensor capable of reflecting the current regional traffic network state is established. The convolution attention mechanism is integrated into a deep reinforcement learning neural network, so that the agent can better focus on the state of vehicles at adjacent intersections. Finally, a multi-agent intersection signal control model based on a deep reinforcement learning algorithm D3QN_CBAM is established. The multi-agent traffic signal control method based on the space-time convolution attention mechanism combines the attention mechanism with regional signal control, and improves the performance and effect of regional traffic signal cooperative control.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically a multi-agent traffic signal control method based on a spatiotemporal convolutional attention mechanism. Background Technology

[0002] Currently, multi-agent signal control can be divided into three categories: independent, centralized, and cooperative. Independent multi-agent algorithms directly train each agent without any cooperation between them. The drawback of this approach is that the behavior of other agents is entirely considered part of the environment; agents only focus on the environmental information of their respective intersections, and the dynamic randomness of other agents' policies is not considered when learning from the local agent. Centralized control can be understood as a single agent jointly controlling all traffic lights within an area, managing intersections within that area. Cooperative control can be achieved through shared states or the introduction of heuristic operational decisions. In the local agent's observation of the environment, the states of some neighboring agents are incorporated, and the value of each action is evaluated based on the cooperative state.

[0003] To improve the model's sensitivity to traffic conditions and enable agents to make better action decisions, we often need to improve its network structure. This helps the model pay more attention to the distribution and dynamics of vehicles near intersections. For multi-agent signal control models, the main problems are as follows: 1. Independent signal control only considers the local traffic environment during training, and the environment within intersection lanes is complex, making it difficult to correctly identify the impact of vehicle states at different times and spaces on traffic flow; 2. Centralized signal control experiences a sharp increase in the state-action space as the road network scales, consuming a large amount of resources; 3. As the number of intersections increases exponentially, the amount of communication information that needs to be transmitted increases significantly. Therefore, it is necessary to ensure effective communication while reducing the amount of transmitted information.

[0004] Therefore, in traffic signal control, there is a problem that signals at adjacent intersections are difficult to coordinate efficiently. In view of the above situation, there is an urgent need to develop a multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism to overcome the shortcomings in current practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-agent traffic signal control method based on a spatiotemporal convolutional attention mechanism to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism includes the following steps:

[0008] Step 1: Model the multi-intersection traffic network in the region as a multi-agent system, defining the state, action, and reward;

[0009] Step 2: Use a convolutional attention network to preprocess the state. By introducing two sub-modules, channel attention and spatial attention, the channel dimension and spatial dimension of the state feature map are weighted respectively to highlight important features and suppress redundant information.

[0010] Step 3: Design a neural network model for intersection traffic light control based on a deep reinforcement learning algorithm with spatiotemporal attention mechanism. The model includes two convolutional layers and a CBAM module, which is used to enhance important features in the feature map.

[0011] Step 4: Train a multi-intersection signal coordination control model using a multi-agent cooperative control algorithm. When agents interact with the environment, the influence of the state and reward of neighboring agents is considered to generate the best action response to the current global state.

[0012] As a further aspect of the present invention: In step 1, the definition of the traffic state at the regional intersection includes:

[0013] The traffic environment is viewed as a traffic grid composed of multiple intersections. Each intersection has four directions, and the edge of each direction consists of four lanes: a straight lane, a left-turn lane, and a right-turn lane.

[0014] The state space of each intersection is defined as follows:

[0015] ;

[0016] in, Indicates intersections in the road network i The lane l The position matrix of vehicles in the middle, Indicates an intersection i The lane l Speed ​​matrix of vehicles in the middle; current intersection i The joint state space consists of the current agent's local state space and the state spaces of neighboring agents, and can be represented as:

[0017] ;

[0018] in, Indicates time t Local status information of the intersection; A set representing the status information of adjacent intersections; It is the set of neighboring nodes.

[0019] As a further aspect of the present invention: in step 1, the action space includes four phase spaces, namely north-south straight right turn, north-south left turn, east-west straight right turn, and east-west left turn.

[0020] As a further aspect of the present invention: In step 1, the reward function is defined as a multi-objective optimization reward, expressed as:

[0021] ;

[0022] in, This represents the total length of the queue of vehicles in all lanes at the intersection at the previous moment; This represents the total number of vehicles waiting in all lanes of the intersection at the previous moment; In order to be in t At any moment, in the lane j The red light duration is C, where C is the maximum red light duration that a driver can tolerate.

[0023] As a further aspect of the present invention: In step 1, a weighted sum is used to combine the agent's reward with the rewards of neighboring domains. A spatial discount factor is introduced so that the reward value of an agent in the neighboring domain is positively correlated with the distance between two agents; the closer the distance, the greater the reward value of the neighboring agent. The spatial discount factor is expressed as:

[0024] ;

[0025] in, Represents a node i and nodes j The distance between them ; It is the discount factor;

[0026] The agent reward function is defined as the sum of the reward obtained in the local environment at the current moment and the rewards of all agents in the neighborhood, expressed as:

[0027] ;

[0028] in, The agent receives a local reward; It is the neighboring agents obtained by the agent. j Reward information, This is the spatial discount factor.

[0029] As a further aspect of the present invention: the calculation formula for the channel attention mechanism is as follows:

[0030] ;

[0031] in, and These represent the number of average pooling and max pooling operations in the channel dimension, respectively. and Indicates the weight parameters of the shared multilayer perceptron; This represents the Sigmoid activation function;

[0032] The formula for calculating spatial attention mechanism is:

[0033] ;

[0034] in, and These represent the average pooling and max pooling operations in the spatial dimension, respectively. This indicates that the kernel size is 7×7; This represents the Sigmoid activation function.

[0035] As a further aspect of the present invention: in step 3, the network structure of the deep reinforcement learning algorithm includes:

[0036] The first convolutional layer is used to extract features of normalized vehicle speed and vehicle position information.

[0037] The CBAM module is used to perform channel attention and spatial attention processing on feature maps;

[0038] The second convolutional layer is used to further process the feature maps;

[0039] Fully connected layers are used to generate state values ​​and action advantage values.

[0040] As a further aspect of the present invention: In step 4, the process of the multi-agent traffic signal cooperative control method includes:

[0041] Initialize the state space, action space, main network, and target network;

[0042] The agent employs an ε-greedy strategy for decision-making, balancing exploration and exploitation;

[0043] After the agent performs the selected action, the environment provides corresponding information and stores the experience sequence in the Replay buffer;

[0044] Based on the priority experience replay strategy, experience data is extracted from the experience replay pool, the loss function is calculated, and the main network parameters are updated.

[0045] As a further aspect of the present invention: in step 4, the formula for calculating the loss function is:

[0046] ;

[0047] in, As a reward value, As a discount factor, The action-value function of the main network. For the parameters of the target network, Indicates the online network is in a state. The action corresponding to the highest output action value. Indicates the target network in state The action value is output based on the action selected by the online network.

[0048] A multi-agent traffic signal control system based on a spatiotemporal convolutional attention mechanism includes a traffic signal control module for executing the aforementioned multi-agent traffic signal dynamic cooperative control method based on a spatiotemporal convolutional attention mechanism, and further includes:

[0049] The traffic simulation environment module is used to simulate multi-intersection traffic networks and provide traffic flow data and status information;

[0050] The intelligent agent module is used to interact with the traffic simulation environment module, execute signal control strategies, and collect feedback information.

[0051] The experience playback module is used to store experience data from the interaction between the intelligent agent and the traffic simulation environment, and is used to train the traffic signal control module.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] 1. This invention addresses the regional traffic signal control problem by proposing a multi-agent traffic signal dynamic cooperative control method based on a spatiotemporal convolutional attention mechanism. This method combines the attention mechanism with regional traffic signal control, thereby improving the performance and effectiveness of regional traffic signal cooperative control.

[0054] 2. Previous traffic signal control based on deep reinforcement learning algorithms only mechanically defined the state of vehicles in the lanes of intersections, and could not better pay attention to the state of vehicles with strong correlations. However, this invention incorporates a convolutional attention mechanism after each convolutional layer, which can better pay attention to the state of vehicles near the intersection, making the signal control strategy more flexible and proactive.

[0055] 3. In multi-agent signal control, in order to address the problems of relatively simple environmental interaction and high state space utilization in independent control and centralized control in regional signal control, the multi-agent dynamic cooperative control method proposed in this invention can consider the influence of the state and reward of neighboring agents when agents interact with the environment, generate the best action response to the current state, and improve the algorithm's coordinated control effect. Attached Figure Description

[0056] Figure 1This is a schematic diagram of multi-agent modeling in a regional road network in an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the operating space of the traffic light in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of the network structure of the D3QN_CBAM algorithm in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of a multi-agent traffic signal cooperative control framework in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0062] Please see Figures 1-4 The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism provided in this invention specifically includes the following steps:

[0063] Step 1: Model the multi-intersection traffic network in the region as a multi-agent system, defining the state, action, and reward;

[0064] Step 2: Use a convolutional attention network to preprocess the state. By introducing two sub-modules, channel attention and spatial attention, the channel dimension and spatial dimension of the state feature map are weighted respectively to highlight important features and suppress redundant information.

[0065] Step 3: Design a neural network model for intersection traffic light control based on the deep reinforcement learning algorithm (D3QN_CBAM) with spatiotemporal attention mechanism. The model includes two convolutional layers and a CBAM module, which is used to enhance important features in the feature map.

[0066] Step 4: Train a multi-intersection signal coordination control model using a multi-agent cooperative control algorithm. When agents interact with the environment, the influence of the state and reward of neighboring agents is considered to generate the best action response to the current global state.

[0067] This invention presents a multi-agent traffic light coordination control method based on a spatiotemporal convolutional attention mechanism and deep reinforcement learning. First, the traffic network of multiple intersections in a region is modeled as a multi-agent system. Each agent, during strategy learning, simultaneously considers the influence of actions by neighboring agents at nearby times, enabling coordinated traffic light control at multiple intersections and establishing a tensor that reflects the current state of the regional traffic network. Second, the convolutional attention mechanism is integrated into a deep reinforcement learning neural network, allowing agents to better monitor the state of vehicles at adjacent intersections. Finally, a multi-agent intersection signal control model based on the deep reinforcement learning algorithm D3QN_CBAM is established. This model is trained using a multi-agent cooperative control algorithm, and the actual traffic state information of the current intersection and neighboring intersections is input into the neural network model to obtain a signal control scheme that improves current traffic flow indicators.

[0068] In one embodiment of the present invention, please refer to Figures 1-4 In step 1, the definition of traffic status at regional intersections includes:

[0069] The traffic environment is viewed as a traffic grid composed of multiple intersections. Each intersection has four directions, and the edge of each direction consists of four lanes: a straight lane, a left-turn lane, and a right-turn lane.

[0070] The state space of each intersection is defined as follows:

[0071] ;

[0072] in, Indicates intersections in the road network i The lane l The position matrix of vehicles in the middle, Indicates an intersection i The lane l Speed ​​matrix of vehicles in the current intersection. i The joint state space consists of the current agent's local state space and the state spaces of neighboring agents, and can be represented as:

[0073] ;

[0074] in, Indicates time t Local status information of the intersection; A set representing the status information of adjacent intersections; It is the set of neighboring nodes.

[0075] In step 1, the action space includes four phase spaces: North-South Straight Right Turn (NSG), North-South Left Turn (NSLG), East-West Straight Right Turn (EWG), and East-West Left Turn (EWLG).

[0076] In step 1, the reward function is defined as the multi-objective optimization reward, expressed as:

[0077] ;

[0078] in, This represents the total length of the queue of vehicles in all lanes at the intersection at the previous moment; This represents the total number of vehicles waiting in all lanes of the intersection at the previous moment; In order to be in t At any moment, in the lane j The red light duration is C, where C is the maximum red light duration that a driver can tolerate.

[0079] In step 1, a weighted sum is used to combine the agent's reward with the rewards from neighboring domains. A spatial discount factor is introduced so that the reward value of an agent in the neighborhood is positively correlated with the distance between two agents; the closer the distance, the greater the reward value obtained from neighboring agents. The spatial discount factor is expressed as:

[0080] ;

[0081] in, Represents a node i and nodes j The distance between them ; It is the discount factor;

[0082] The agent reward function is defined as the sum of the reward obtained in the local environment at the current moment and the rewards of all agents in the neighborhood, expressed as:

[0083] ;

[0084] in, The agent receives a local reward; It is the neighboring agents obtained by the agent. j Reward information, This is the spatial discount factor.

[0085] In one embodiment of the present invention, please refer to Figures 1-4 In step 2, the calculation formula for the channel attention mechanism is:

[0086] ;

[0087] in, and These represent the number of average pooling and max pooling operations in the channel dimension, respectively. and This represents the weight parameters of the shared multilayer perceptron. This represents the Sigmoid activation function;

[0088] The formula for calculating spatial attention mechanism is:

[0089] ;

[0090] in, and These represent the average pooling and max pooling operations in the spatial dimension, respectively. This indicates that the kernel size is 7×7. This represents the Sigmoid activation function.

[0091] Please see Figures 1-4 In step 3, the network structure of the D3QN_CBAM algorithm includes:

[0092] The first convolutional layer is used to extract features of normalized vehicle speed and vehicle position information.

[0093] The CBAM module is used to perform channel attention and spatial attention processing on feature maps;

[0094] The second convolutional layer is used to further process the feature maps;

[0095] Fully connected layers are used to generate state values ​​and action advantage values.

[0096] Please see Figures 1-4 In step 4, the process of the multi-agent traffic signal cooperative control method includes:

[0097] Initialize the state space, action space, main network, and target network;

[0098] The agent employs an ε-greedy strategy for decision-making, balancing exploration and exploitation;

[0099] After the agent performs the selected action, the environment provides corresponding information and stores the experience sequence in the Replay buffer;

[0100] Based on the priority experience replay strategy, experience data is extracted from the experience replay pool, the loss function is calculated, and the main network parameters are updated.

[0101] In step 4, the formula for calculating the loss function is:

[0102] ;

[0103] in, As a reward value, As a discount factor, The action-value function of the main network. For the parameters of the target network, Indicates the online network is in a state. The action corresponding to the highest output action value. Indicates the target network in state The action value is output based on the action selected by the online network.

[0104] This invention presents a multi-agent traffic signal control method based on a spatiotemporal convolutional attention mechanism. It optimizes the network structure by incorporating a convolutional attention mechanism. An attention mechanism is added after each convolutional layer of the initial network, enabling better monitoring of the state of vehicles near intersections, thus making the signal control strategy more flexible and proactive. The improved network is then applied to a multi-agent road network to achieve cooperative control. The method includes the following steps:

[0105] Step 1: Model the multi-intersection traffic network of a region as a multi-agent system, defining its state, actions, and rewards. The steps for defining the traffic state at regional intersections are as follows:

[0106] (1) As Figure 1 As shown, the traffic environment is viewed as a traffic grid consisting of multiple intersections. Each intersection has four directions. The edge of each direction consists of four lanes: a straight lane, a left-turn lane, and a right-turn lane. In a given traffic grid, each intersection has an agent that controls the traffic signals, and n1, n2, n3, and n4 are its neighboring intersections. Each agent has at least two neighbors and at most four neighbors. i With any adjacent intersection j , .

[0107] In a multi-intersection environment, the state definition space of intersection i is defined as follows:

[0108] (1);

[0109] in, Indicates intersections in the road network i The lane l The position matrix of vehicles in the middle, Indicates an intersection i The lane l Speed ​​matrix of vehicles in the current intersection. i The joint state space consists of the current agent's local state space and the state spaces of neighboring agents, and can be represented as:

[0110] (2);

[0111] in, Indicates time t Local status information of the intersection; A set representing the status information of adjacent intersections; It is the set of neighboring nodes as shown in equation (1).

[0112] (2) Action space such as Figure 2 As shown, it contains four phase spaces, from left to right: North-South Straight Right Turn (NSG), North-South Left Turn (NSLG), East-West Straight Right Turn (EWG), and East-West Left Turn (EWLG).

[0113] (3) Multi-objective optimization reward is used as the reward representation for the reinforcement learning algorithm. The reward function is defined as:

[0114] (3);

[0115] in, This represents the total length of the queue of vehicles in all lanes at the intersection at the previous moment; This represents the total number of vehicles waiting in all lanes of the intersection at the previous moment; In order to be in t At any moment, in the lane j The red light duration is denoted by C, where C is the maximum tolerable red light time for the driver. If the red light duration for all lanes is less than the maximum tolerable time, the penalty term will be zero. If some lanes have red light durations exceeding the maximum tolerable time, a penalty will be imposed.

[0116] A weighted sum is used to combine the agent's reward with the rewards from neighboring domains. A spatial discount factor is introduced so that the reward value of an agent in the neighborhood is positively correlated with the distance between two agents; the closer the distance, the greater the reward value from neighboring agents. The spatial discount factor is expressed as:

[0117] (4);

[0118] in, Represents a node i and nodes j The distance between them ; It is the discount factor.

[0119] The agent reward function is defined as the sum of the reward obtained in the local environment at the current moment and the rewards of all agents in the neighborhood, expressed as:

[0120] (5);

[0121] in, The agent receives a local reward; It is the neighboring agents obtained by the agent. j Reward information, This is the spatial discount factor.

[0122] Step 2: Use a convolutional attention network to preprocess the state. The model introduces two sub-modules, channel attention and spatial attention, to weight the channel dimension and spatial dimension of the state feature map respectively, thereby effectively highlighting important features and suppressing redundant information.

[0123] This invention integrates channel information into the feature map by applying average pooling and max pooling operations, and uses a shared multilayer perceptron to generate channel attention maps individually. These generated attention maps are added element-wise, and the weight information of each channel is obtained using the Sigmoid function. The calculation formula (6) is as follows:

[0124] (6);

[0125] in, and These represent the number of average pooling and max pooling operations in the channel dimension, respectively. and This represents the weight parameters of the shared multilayer perceptron. This represents the Sigmoid activation function.

[0126] To calculate spatial attention, this invention first applies average pooling and max pooling operations along the spatial dimension to obtain the spatial average and maximum values ​​at each spatial location. These averages and maximum values ​​are then combined by concatenation. Next, convolution operations are performed to generate a 3D spatial attention map. Finally, the sigmoid function is used to obtain the weight information. The calculation formula (7) is as follows:

[0127] (7);

[0128] in, and These represent the average pooling and max pooling operations in the spatial dimension, respectively. This indicates that the kernel size is 7×7. This represents the Sigmoid activation function.

[0129] Step 3: Design a neural network model for intersection traffic light control using a deep reinforcement learning algorithm based on spatiotemporal attention mechanism (DuelingDoubleDeepQNetworkwithCBAM, D3QN_CBAM), such as... Figure 3 As shown.

[0130] D3QN_CBAM allows the agent to automatically focus on key aspects of a state (such as vehicles approaching an intersection) without being distracted by noise or irrelevant features. The state in this system consists of two parts: normalized vehicle speed information and vehicle position information. First, the normalized vehicle speed and position information are fed into the first convolutional layer to extract relevant features, generating a feature map; this feature map is then input into the CBAM module. In the CBAM module, the feature map undergoes channel attention and spatial attention processing, enabling the model to focus more on important features. The channel attention mechanism enhances the features in each channel, while the spatial attention mechanism captures important information from different locations.

[0131] The feature maps processed by the CBAM module are passed to a second convolutional layer for further processing. The features obtained through convolution are again fed into the CBAM module to enhance the attention feature weights. The output data is then flattened and concatenated with phase state information. Finally, this concatenated data is processed through two fully connected layers and a ReLU operation to generate an output for two subsequent branches. The first branch computes the state values, while the second branch computes the action advantage values; for each operation, its dimension matches the dimension of the output layer.

[0132] Step 4: The flowchart of the multi-agent traffic signal cooperative control method is as follows: Figure 4 As shown.

[0133] The state space, action space, main network, and target network are initialized, along with their structures and parameters. Additionally, the capacity of the experience replay pool and training pool needs to be initialized. During the interaction between the agent and its environment, the agent first acquires the current state of the environment. Status information of neighboring intersections To form a joint state The input is fed into the neural network.

[0134] Intelligent agents adopt ε The -greedy strategy is used for decision-making, balance exploration, and utilization. After taking an action, a semaphore returns a reward value. Each reward value is saved and calculated before starting the next iteration. After the agent executes the chosen action, the environment provides feedback based on the result. Specifically, the agent receives the state... Select operation In execution During the action, the active phase of a cycle is activated first, and then the reward is obtained. and enter a new state. and the experience sequence It is stored in the Replay buffer.

[0135] Based on the priority-based experience replay strategy, extract from the experience replay pool. B Using empirical data, the loss function is calculated using formula (8). .

[0136] (8);

[0137] in, As a reward value, As a discount factor, The action-value function of the main network. These are the parameters of the target network;

[0138] Subsequently, the parameters of the main network are updated using TD-error, and the priority of the transition sequences is adjusted simultaneously. After a fixed number of iterations of training, the parameters in the target network are updated synchronously. .

[0139] In one embodiment of the present invention, a multi-agent traffic signal control system based on a spatiotemporal convolutional attention mechanism includes a traffic signal control module for executing the aforementioned multi-agent traffic signal dynamic cooperative control method based on a spatiotemporal convolutional attention mechanism, and further includes:

[0140] The traffic simulation environment module is used to simulate multi-intersection traffic networks and provide traffic flow data and status information;

[0141] The intelligent agent module is used to interact with the traffic simulation environment module, execute signal control strategies, and collect feedback information.

[0142] The experience playback module is used to store experience data from the interaction between the intelligent agent and the traffic simulation environment, and is used to train the traffic signal control module.

[0143] In summary, this invention, through a multi-agent traffic light coordination control method based on spatiotemporal convolutional attention and deep reinforcement learning, enables the local agent to consider the dynamic changes and responses of neighboring agents. Agents can exchange state information, and a spatiotemporal convolutional attention mechanism is introduced to better monitor the states of vehicles at adjacent intersections. When interacting with the environment, the agent considers the influence of the states and rewards of neighboring agents. The action generated after reaching Nash equilibrium through Markov game theory can be represented as the optimal action response to the current global state. During the decision-making process, the agent improves the algorithm's coordination control effect by sharing state and reward information with neighboring intersections.

[0144] The method was validated in the SUMO traffic simulation system, including two scenarios (peak traffic flow and off-peak traffic flow), as detailed below:

[0145] Table 1 Evaluation metrics for each algorithm under low peak flow conditions

[0146]

[0147] In off-peak traffic scenarios, compared to the D3QN algorithm, the average queue length was reduced by 19.6%, the average vehicle waiting time was reduced by 22.6%, and the average driving speed was increased by 8.8%.

[0148] Table 2 Evaluation metrics for each algorithm under peak flow conditions

[0149]

[0150] In high-traffic scenarios, compared to the D3QN algorithm, the average queue length was reduced by 17.8%, the average vehicle waiting time was reduced by 39.6%, and the average driving speed was increased by 17.2%.

[0151] Experimental results show that, under the same traffic flow conditions, the improved algorithm outperforms the D3QN and Webster algorithms in terms of both control effectiveness and convergence speed.

[0152] It should be noted that, in this invention, although the specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism, characterized in that, Specifically, the following steps are included: Step 1: Model the multi-intersection traffic network in the region as a multi-agent system, defining the state, action, and reward; Step 2: Use a convolutional attention network to preprocess the state. By introducing two sub-modules, channel attention and spatial attention, the channel dimension and spatial dimension of the state feature map are weighted respectively to highlight important features and suppress redundant information. Step 3: Design a neural network model for intersection traffic light control based on a deep reinforcement learning algorithm with spatiotemporal attention mechanism. The model includes two convolutional layers and a CBAM module, which is used to enhance important features in the feature map. Step 4: Train a multi-intersection signal coordination control model through a multi-agent cooperative control algorithm. When the agent interacts with the environment, it considers the influence of the state and reward of the neighboring agents and generates the best action response to the current global state. The formula for calculating the channel attention mechanism is: ； in, and These represent the number of average pooling and max pooling operations in the channel dimension, respectively. and Indicates the weight parameters of the shared multilayer perceptron; This represents the Sigmoid activation function; The formula for calculating spatial attention mechanism is: ； in, and These represent the average pooling and max pooling operations in the spatial dimension, respectively. This indicates that the kernel size is 7×7; This represents the Sigmoid activation function; In step 4, the process of the multi-agent traffic signal cooperative control method includes: Initialize the state space, action space, main network, and target network; The agent employs an ε-greedy strategy for decision-making, balancing exploration and exploitation; After the agent performs the selected action, the environment provides corresponding information and stores the experience sequence in the Replay buffer; Based on the priority experience replay strategy, experience data is extracted from the experience replay pool, the loss function is calculated, and the main network parameters are updated. In step 4, the formula for calculating the loss function is: ； in, As a reward value, As a discount factor, The action-value function of the main network. For the parameters of the target network, Indicates the online network is in a state. The action corresponding to the highest output action value. Indicates the target network in state The action value is output based on the action selected by the online network.

2. The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism according to claim 1, characterized in that, In step 1, the definition of traffic status at regional intersections includes: The traffic environment is viewed as a traffic grid composed of multiple intersections. Each intersection has four directions, and the edge of each direction consists of four lanes: a straight lane, a left-turn lane, and a right-turn lane. The state space of each intersection is defined as follows: ； in, Indicates intersections in the road network i The lane l The position matrix of vehicles in the middle, Indicates an intersection i The lane l Speed ​​matrix of vehicles in the middle; current intersection i The joint state space consists of the current agent's local state space and the state spaces of neighboring agents, and is represented as: ； in, Indicates time t Local status information of the intersection; A set representing the status information of adjacent intersections; It is the set of neighboring nodes.

3. The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism according to claim 2, characterized in that, In step 1, the action space includes four phase spaces: north-south straight right turn, north-south left turn, east-west straight right turn, and east-west left turn.

4. The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism according to claim 3, characterized in that, In step 1, the reward function is defined as the multi-objective optimization reward, expressed as: ； in, This represents the total length of the queue of vehicles in all lanes at the intersection at the previous moment; This represents the total number of vehicles waiting in all lanes of the intersection at the previous moment; In order to be in t At any moment, in the lane j The red light duration is C, where C is the maximum red light duration that a driver can tolerate.

5. The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism according to claim 4, characterized in that, In step 1, a weighted sum is used to combine the agent's reward with the rewards from neighboring domains. A spatial discount factor is introduced so that the reward value of an agent in the neighborhood is positively correlated with the distance between two agents; the closer the distance, the greater the reward value obtained from neighboring agents. The spatial discount factor is expressed as: ； in, Represents a node i and nodes j The distance between them ; It is the discount factor; The agent reward function is defined as the sum of the reward obtained in the local environment at the current moment and the rewards of all agents in the neighborhood, expressed as: ； in, The agent receives a local reward; It is the neighboring agents obtained by the agent. j Reward information, This is the spatial discount factor.

6. The multi-agent traffic signal control method based on spatiotemporal convolutional attention mechanism according to claim 1, characterized in that, In step 3, the network structure of the deep reinforcement learning algorithm includes: The first convolutional layer is used to extract features of normalized vehicle speed and vehicle position information. The CBAM module is used to perform channel attention and spatial attention processing on feature maps; The second convolutional layer is used to further process the feature maps; Fully connected layers are used to generate state values ​​and action advantage values.

7. A multi-agent traffic signal control system based on a spatiotemporal convolutional attention mechanism, comprising a traffic signal control module for executing the multi-agent traffic signal dynamic cooperative control method based on a spatiotemporal convolutional attention mechanism according to any one of claims 1-6, characterized in that, Also includes: The traffic simulation environment module is used to simulate multi-intersection traffic networks and provide traffic flow data and status information; The intelligent agent module is used to interact with the traffic simulation environment module, execute signal control strategies, and collect feedback information. The experience playback module is used to store experience data from the interaction between the intelligent agent and the traffic simulation environment, and is used to train the traffic signal control module.

Images