Multi-agent traffic signal control method based on hierarchical contrastive learning

Abstract

The application provides a multi-agent traffic signal control method based on hierarchical contrast learning, which constructs a traffic simulation environment according to a traffic road network, and models each intersection in the traffic simulation environment as an independent agent; the agent interacts with a SUMO simulation environment, collects traffic state information, signal control actions, instant rewards and next state traffic state information of each intersection, and stores them in an experience replay buffer; the agent is grouped to generate regional pseudo labels; according to different regional pseudo labels, regional division is carried out, a subgraph is constructed, and intra-regional feature aggregation is carried out on the subgraph to realize fine credit allocation of structure perception; finally, joint optimization is carried out based on contrast learning and a QTRAN framework to realize efficient multi-agent collaborative control. The application can provide an optimal signal timing strategy for each intersection, improve road network traffic efficiency, and significantly reduce vehicle queue length and waiting time.

Description

Technical Field

[0001] This invention relates to the field of traffic data processing technology, and in particular to a multi-agent traffic signal control method based on hierarchical contrastive learning. Background Technology

[0002] In recent years, with the continuous expansion of megacities, traffic congestion has become a core problem restricting sustainable urban development. As key nodes where traffic flows from different directions converge, signalized intersections are generally considered a major bottleneck to the operational efficiency of urban road networks. Frequent starting, stopping, accelerating, and decelerating of vehicles at intersections not only leads to decreased traffic efficiency but also significantly contributes to energy waste and increased exhaust pollution. Therefore, developing intelligent signal control methods that can adapt to dynamic traffic environments is of great significance for improving road network capacity and promoting green transportation development.

[0003] However, with the increasing complexity of urban road networks, traditional timed control and sensor-based control methods perform poorly in dealing with large-scale dynamic traffic flows, often exacerbating congestion due to rigid control strategies. The rise of deep reinforcement learning technology has provided a new solution for traffic signal control. Existing research mainly focuses on single-agent frameworks, achieving some success in isolated intersection scenarios, but faces challenges such as joint state space dimensionality explosion and difficulties in agent cooperation when extended to large-scale road networks. While the introduction of multi-agent reinforcement learning has partially alleviated these problems, it still has significant limitations: the commonly used parameter sharing mechanism leads to homogenization of agent strategies at different intersections, making it difficult to adapt to the heterogeneous characteristics of intersections in different areas; at the same time, existing credit allocation methods fail to fully consider the road network topology and cannot accurately evaluate the contribution of individual agent decisions to the overall system performance. These problems severely restrict the control effect of multi-agent systems in large-scale urban road networks, urgently requiring breakthroughs through innovative methods. Summary of the Invention

[0004] In view of the above, the main objective of this invention is to propose a multi-agent traffic signal control method based on hierarchical contrastive learning to solve the aforementioned technical problems.

[0005] This invention proposes a multi-agent traffic signal control method based on hierarchical contrastive learning, the method comprising the following steps:

[0006] Step 1: Construct a traffic simulation environment based on the traffic network, and model each intersection in the traffic simulation environment as an independent intelligent agent;

[0007] The intelligent agent interacts with the SUMO simulation environment, collecting traffic state information, signal control actions, instant rewards, and traffic state information for the next state at each intersection, and storing them in the experience playback buffer.

[0008] Step 2: Extract the latent representation of traffic state information using the policy network, group the latent representations of all agents using the K-means clustering algorithm to generate cluster labels, and use the cluster labels as pseudo-labels for different regions; divide all intersections into different functional regions according to the different regional pseudo-labels, construct a sub-graph for each functional region, and perform regional feature aggregation on the sub-graphs to obtain cluster-level state vectors.

[0009] Step 3: Using different regional pseudo-labels as supervision signals, construct a contrastive learning loss function; using cluster-level state vectors as the abstract representation of traffic state in the corresponding functional areas, construct a QTRAN total loss function based on the QTRAN framework; and construct a joint loss function using the contrastive learning loss function and the QTRAN total loss function.

[0010] Step 4: Sample small batches of data from the experience replay buffer, use the Adam optimizer to jointly update the parameters of the policy network and the value network to minimize the joint loss function, obtain the trained policy network, and deploy the trained policy network to each intersection agent to realize distributed signal control.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] 1. This invention introduces an unsupervised contrastive learning mechanism in the form of pseudo-labels, and applies structured constraints at the representation level of the agent policy network, enabling the agent to generate differentiated policies according to the feature cluster of its region. This effectively solves the problem of policy homogenization in multi-agent systems and significantly improves the model's adaptability and generalization ability in heterogeneous traffic scenarios.

[0013] 2. This invention optimizes the credit allocation process of the QTRAN algorithm by dividing the global road network into multiple cooperative sub-regions and using graph convolutional networks to model spatial dependencies within the sub-regions. This enables global rewards to be more accurately associated with local behaviors, significantly improving the efficiency and stability of multi-agent cooperative control.

[0014] 3. The present invention innovatively combines the contrastive learning loss function with the QTRAN constraint loss function to achieve a decoupled policy learning and credit allocation optimization path. While maintaining the algorithm training efficiency, it significantly improves the control performance of the model in large-scale heterogeneous road networks, providing an effective solution for signal control in complex urban traffic scenarios.

[0015] 4. The system architecture of this invention adopts a centralized training and distributed execution paradigm, and reduces the global computational complexity from O(N) through a hierarchical graph convolutional structure. 2 Reduced to O(∑|C) k | 2O represents the computational complexity, and C represents the time complexity. k Given the sample set of the k-th cluster, it supports parallel processing of graph convolution operations within each cluster, significantly improving the scalability of the algorithm while ensuring control performance, and providing a feasible technical path for real-time signal control of urban road networks.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multi-agent traffic signal control method based on hierarchical contrastive learning proposed in this invention;

[0018] Figure 2 This is a model framework diagram of a multi-agent traffic signal control method based on hierarchical contrastive learning proposed in this invention;

[0019] Figure 3 A schematic diagram illustrating the diversification of comparison strategies;

[0020] Figure 4 A schematic diagram of the setup for the simulated road network experiment;

[0021] Figure 5 Schematic diagram of simulated road network traffic flow generation;

[0022] Figure 6 This is a schematic diagram of a four-phase intersection in a simulated road network.

[0023] Figure 7 Training curves for four multi-agent deep reinforcement learning models;

[0024] Figure 8 The graph shows the average queue length at intersections for four multi-agent deep reinforcement learning models in a simulated road network. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0026] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0027] Please see Figure 1 and Figure 2 This embodiment provides a multi-agent traffic signal control method based on hierarchical contrastive learning, the method comprising the following steps:

[0028] Step 1: Construct a traffic simulation environment based on the traffic network, and model each intersection in the traffic simulation environment as an independent intelligent agent;

[0029] The intelligent agent interacts with the SUMO simulation environment, collecting traffic state information, signal control actions, instant rewards, and traffic state information for the next state at each intersection, and storing them in the experience playback buffer.

[0030] In a preferred embodiment of the present invention, traffic state information is defined as phase state. Lane space occupancy Lane traffic flow Lane queue length ;in, This indicates the number of phases configured at the intersection. Indicates the number of import lanes. This indicates the number of spatial segments into which each lane is divided;

[0031] Signal control actions are defined as the finite set of phases that an agent, consisting of all combinations of traffic signals from the traffic signal controller, can choose at an intersection; the set of selectable actions for each agent i. For this intersection, there is a finite set of phases; agent i starts from... Selecting and executing a phase action is called the individual action of the agent, denoted as . For a cross-shaped intersection with a left-turn lane in the synthetic road network, the selectable phase set is divided into four combined red-green signal phases: east-west straight and right-turn phases, east-west left-turn and right-turn phases, north-south straight and right-turn phases, and north-south left-turn and right-turn phases. In the synthetic road network constructed in this embodiment, all intersections adopt a standard four-phase control scheme. Therefore, the action set of each agent (intersection) is the same, containing four selectable phases.

[0032] The immediate reward is defined as the reward received by agent i from traffic environment feedback at discrete time step t. This traffic environment feedback includes local congestion relief, individual travel experience, and global supply-demand balance, and the corresponding process has the following relationship:

[0033] ;

[0034] in, Indicates an immediate reward. This indicates the number of vehicles (speeds) waiting behind the stop line in the m-th lane at the intersection entrance. (Time determination), statistical analysis is performed in real time through roadside sensors; Indicates vehicle v The current time step adds a waiting time. Indicates the pressure at the intersection. These represent the weighting coefficients corresponding to the number of vehicles waiting behind the stop line, the newly added waiting time at the current time step, and the intersection pressure, respectively. Indicates the static number of lanes. This represents the dynamic count of vehicles, which changes over time.

[0035] Among them, vehicles j The calculation process for the new waiting time at the current time step follows the following formula:

[0036] ;

[0037] in, This indicates the moment when the system enters a waiting state. Indicates the index of the current time step. Indicates a historical time step. Indicates vehicle v Steps in history The new waiting time, From 1 to An integer of -1.

[0038] Step 2: Extract the latent representation of traffic state information using the policy network, group the latent representations of all agents using the K-means clustering algorithm to generate cluster labels, and use the cluster labels as pseudo-labels for different regions; divide all intersections into different functional regions according to the different regional pseudo-labels, construct a sub-graph for each functional region, and perform regional feature aggregation on the sub-graphs to obtain cluster-level state vectors.

[0039] As a preferred embodiment of the present invention, extracting the potential representation of traffic state information specifically includes the following steps:

[0040] Three independent encoders are set up, including a lane queuing encoder, a bird's-eye view (BEV) encoder, and a time delay encoder.

[0041] The lane queue length is input into the lane queue encoder. The lane queue length is first normalized to obtain... M Observation vector of each lane Then M The observation vectors of each lane are processed through a two-layer fully connected network to obtain lane queuing features. The lane queuing encoder uses a two-layer fully connected network, and the specific formula is as follows:

[0042] ;

[0043] ;

[0044] in, This represents the output feature vector of the first fully connected layer. Represents the linear rectified activation function. Let represent the weight matrix and bias vector of the first fully connected layer, respectively. Let these represent the weight matrix and bias vector of the second fully connected layer, respectively. Indicates lane queuing characteristics, ;

[0045] Lane space occupancy and lane traffic flow are input into the bird's-eye view encoder and reshaped into two three-dimensional tensors. The two three-dimensional tensors are then concatenated along the channel dimension to obtain the bird's-eye view feature map. ; Indicates the number of channels. The spatial dimensions are represented; then the bird's-eye view feature map is input into the ResNet-18 network, and the dimensions are adjusted through a linear layer to obtain the spatial topological features, as shown in the following formula:

[0046] ;

[0047] in, Representing spatial topological features, This represents the ResNet-18 deep residual network; This represents a linear layer, which involves global average pooling of the network output to obtain a vector representation and performing dimension mapping.

[0048] The phase state is input to the time delay encoder. The current phase duration and period progress ratio are extracted from the phase state to obtain the time feature vector. Then, the time feature vector is mapped to a time embedding; the time delay encoder uses a multilayer perceptron, and the specific formula is as follows:

[0049] ;

[0050] in, Indicates time embedding, This represents a multilayer perceptron.

[0051] By fusing lane queuing features, spatial topological features, and temporal embeddings through element-wise addition, a latent representation of traffic state information is obtained, and the corresponding formula has the following relationship:

[0052] ;

[0053] in, Representing a potential representation, This indicates an element-wise addition operation.

[0054] In a preferred embodiment of the present invention, the specific steps for generating the signal control action are as follows:

[0055] The latent representation is input into a gated recurrent unit (GRU) to capture temporal dependencies for time modeling. The corresponding process has the following relationship:

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] in, This represents the Sigmoid activation function; This represents the hidden state at the previous time step, which is the state of the gated loop unit at time step [time step]. t Output -1; These represent the updated weight matrix and bias vector, respectively. This indicates updating the gate vector. This indicates resetting the gate vector. These represent the reset weight matrix and bias vector, respectively. Let these represent the weight matrix and bias vector used to calculate the candidate state, respectively. Represents the candidate hidden state vector. This indicates the hidden state after the current update. This indicates element-wise multiplication;

[0061] The hidden state is represented by the output phase action probability distribution of a fully connected layer, and the corresponding process has the following relationship:

[0062] ;

[0063] in, This represents a policy function whose output is given by the traffic state. Next, select each phase action. The probability distribution, This represents the action performed by the agent at time step t. Indicates at time step t Traffic conditions observed by the intelligent agent Represents the normalized exponential function, These represent the trainable weight matrix and bias vector of the fully connected layer, respectively, representing the probability distribution of the generated actions.

[0064] The phase action probability distribution is sampled to obtain the signal control action.

[0065] In a preferred embodiment of the present invention, the latent representations of all agents are grouped using the K-means clustering algorithm to generate cluster labels, which are then used as pseudo-labels for different regions. The specific steps include:

[0066] potential representation After being processed by a gated recurrent unit, the features are used as input features for the clustering algorithm; where, Indicates the first N The potential representation of an intelligent agent , The dimension of the potential representation;

[0067] Using minimizing the sum of within-cluster squared errors of all clusters as the optimization objective, the K-means clustering algorithm is applied to perform unsupervised clustering of the N latent representations. The process of minimizing the within-cluster squared errors of all clusters is governed by the following relationship:

[0068] ;

[0069] in, This indicates the preset number of clusters. This represents the latent representation of the i-th agent. Let K represent the sample set of the k-th cluster. Cluster The center vector; K The -means clustering algorithm executes by iterating through the following two steps until the convergence condition is met;

[0070] Assignment steps: Calculate the Euclidean distance from each latent representation to the cluster center and assign it to the nearest cluster;

[0071] Update steps: Based on the current cluster assignment results, recalculate the center of each cluster as the mean of all representations within that cluster;

[0072] Label generation: After clustering is completed, the final clustering result is obtained. Based on the final clustering result, a cluster label is assigned to each agent. The cluster label represents the functional area number to which it belongs. The cluster label is used as the pseudo-label of the agent's region for subsequent policy differentiation learning and hierarchical credit allocation.

[0073] In this embodiment, to enhance the latent representation To enhance the discrimination ability, a contrastive learning mechanism is introduced. Based on the intersection morphology, [the following is applied]: N The latent representation of an agent It is the potential representation after being processed by the gated loop unit, and it is executed.K -means clustering, generating pseudo-labels Identify region types. Pseudo-labels serve as self-supervised signals, guiding the contrastive learning loss function so that agents within the same functional region learn similar policies, while agents in different regions learn differentiated policies.

[0074] In a preferred embodiment of the present invention, all intersections are divided into different functional areas based on different regional pseudo-labels. A sub-graph is constructed for each functional area, and regional feature aggregation is performed on the sub-graphs to obtain a cluster-level state vector. Specifically, the steps include the following:

[0075] Based on the pseudo-labels of the regions, all agents are assigned to the corresponding functional regions;

[0076] Using intersection nodes within the current functional area as nodes, the actual road network topology connections or spatial proximity are defined as edges, and the Euclidean distance between intersections and road capacity are defined as edge weights, modeling the system as a subgraph. , ;in This represents the set of all intersection nodes belonging to this functional area. This represents the set of edges defined based on actual road network topology or spatial proximity.

[0077] The calculation process of edge weights follows the following formula:

[0078] ;

[0079] in, Indicates an intersection Intersection Edge weights between them Indicates an intersection Intersection The Euclidean distance between them Indicates road capacity. Indicates the scale parameter;

[0080] The latent representation of each agent is used as the initial node feature of each subgraph node;

[0081] Each subgraph is subjected to a two-layer graph convolutional network for feature propagation and aggregation to capture spatial dependencies within the region, thus obtaining region-aware features. The corresponding process has the following relationship:

[0082] ;

[0083] ;

[0084] in, These represent the trainable weight matrices of the two layers of the GCN. Represents region-aware features. This represents the input feature matrix, where each row in the input feature matrix corresponds to a node in the subgraph. ; This represents an adjacency matrix with self-loops. , The adjacency matrix represents the subgraph. Represents the identity matrix. express The degree matrix;

[0085] Max pooling is performed on all region-sensing features to generate the cluster-level state vector for that functional region. The corresponding process has the following relationship:

[0086] ;

[0087] in, Represents the cluster-level state vector. This represents the dimension of the cluster-level state vector.

[0088] Step 3: Using different regional pseudo-labels as supervision signals, construct a contrastive learning loss function; using cluster-level state vectors as the abstract representation of traffic state in the corresponding functional areas, construct a QTRAN total loss function based on the QTRAN framework; and construct a joint loss function using the contrastive learning loss function and the QTRAN total loss function.

[0089] To further improve the learning performance of the agents, a contrastive learning enhancement component is introduced into the algorithm. Each agent i generates a latent representation. An embedding vector is used to represent the current state of each agent. Then, K-means clustering is used to analyze the latent representations of all agents. Perform clustering and generate pseudo-labels During model training, a batch of data is sampled from the experience replay buffer each time, containing the latent representations of N agents and their corresponding pseudo-labels. Based on these pseudo-labels, a contrastive learning loss function is constructed to narrow the distance between the latent representations of agents within the same functional region (with the same pseudo-label). These pseudo-labels represent the category of the environmental state in which each agent exists. Contrastive learning loss function. The calculation is achieved by measuring the similarity between latent representations. The specific formula for the contrastive learning loss function is as follows:

[0090] ;

[0091] in, Indicates batch size, This represents the latent representation of a positive sample that has the same pseudo-label in the same region (i.e., belongs to the same cluster). Let i and j represent the latent representations of the i-th and j-th agents, respectively. This indicates the cosine similarity calculation operation. Indicates samples within the same cluster; This represents a temperature parameter used to adjust the smoothness of the distribution.

[0092] The aforementioned contrastive learning mechanism guides model learning by using pseudo-labels generated through clustering as supervisory signals. Its core design lies in achieving policy diversification: bringing agents within the same functional region closer together, encouraging them to learn coordinated and similar policies; simultaneously pushing agents from different functional regions further apart, encouraging them to learn differentiated policies based on their respective regional traffic characteristics. A schematic diagram of this contrastive policy diversification design is shown below. Figure 3 As shown.

[0093] In a preferred embodiment of the present invention, cluster-level state vectors are used as abstract representations of traffic states in corresponding functional areas. Based on the QTRAN framework, a QTRAN total loss function is constructed, specifically including the following steps:

[0094] Global Fusion and QTRAN Constraint Implementation: K Cluster-Level State Vectors By fusing two independent fully connected networks to output a QTRAN function, the corresponding process has the following relationship:

[0095] potential representation Inputting a two-layer fully connected network to calculate the individual action value function of each agent at the intersection, the corresponding process has the following relationship:

[0096] ;

[0097] ;

[0098] in, Indicates hidden features in the middle. This represents the trainable parameters of a fully connected network. , Represents the local state dimension; The function representing the value of an individual action of agent i; This represents the Q-value of all possible phase actions of agent i, where each action... The individual action value function is expressed as: .

[0099] Using cluster-level state vectors as abstract representations of traffic states in corresponding functional areas, K Cluster-level state vectors By fusing two independent fully connected networks, the joint action value function and state value function are calculated. The corresponding process has the following relationship:

[0100] ;

[0101] ;

[0102] in, This represents the vectorized encoding of the joint action 'a', which consists of the individual actions of all agents. Let represent the weights and biases of the joint action value function, respectively. Let these represent the weights and biases of the state-value function, respectively. This represents vector concatenation. This represents the value function of the joint action. This represents the expected cumulative return under the global state and joint actions. Representing the global state and joint action respectively; Represents the state value function. This represents the expected return in the global state;

[0103] Based on the QTRAN framework, the QTRAN total loss function is constructed using individual action value functions, joint action value functions, and state value functions.

[0104] The calculation process of the QTRAN total loss function follows the following relationship:

[0105] The QTRAN total loss function includes the time difference loss function, the optimal action constraint loss function, and the non-optimal action constraint loss function, and the corresponding processes have the following relationship:

[0106] ;

[0107] in, Represents the total loss function of QTRAN; Indicates the next global state; Represents global experience replay samples The reward item in the table represents the experience replay sample at discrete time steps. t Instant rewards generated ,Right now ; Parameters representing online networks; Represents the time difference loss function. This represents the optimal action constraint loss function. This represents the loss function for non-optimal action constraints. These represent the weights corresponding to the optimal action constraint loss function and the non-optimal action constraint loss function, respectively.

[0108] Time difference loss function ( ): Used to train a joint action value network to accurately approximate the Bellman objective value;

[0109] ;

[0110] ;

[0111] in, This indicates the combined action value of online network output; Indicates the discount factor; This represents the optimal combination of actions for the individual in the next state. This represents the optimal combination of individual actions chosen by the target network in the next global state, based on its policy. The estimated value; These represent the parameters of the target network. Indicates the target value;

[0112] Optimal action constraint loss function ( ): Ensure that when all agents take their individual optimal actions, the joint action value equals the sum of the individual action values ​​of each agent minus the state baseline;

[0113] ;

[0114] in, Represents the state-value function, outputting the state. The baseline value; A fixed estimate representing the sum of the values ​​of individual actions is defined as follows: That is, when calculating the optimal action constraint loss function, the value is fixed. It does not participate in gradient updates; This represents the optimal action of agent i. Q value, The optimal action of agent i is calculated by the target network. This represents the optimal action combination for the current individual; , Let N be the individual optimal action of agent N, where N is the optimal action of agent N. ;

[0115] Non-optimal action constraint loss function ): Ensures that for any non-optimal joint action a The combined value of their actions shall not exceed the sum of their individual values ​​minus the baseline.

[0116] ;

[0117] in, B This represents a set of non-optimal joint actions sampled from experience; Represents a linear rectified function; Indicates the corresponding action a A fixed estimate of the sum of individual values, i.e., a fixed value when calculating the non-optimal action constraint loss function. It does not participate in gradient updates; , Represents the action of agent i The Q value is calculated from the target network.

[0118] In a preferred embodiment of the present invention, a joint loss function is constructed using the contrastive learning loss function and the QTRAN total loss function. The corresponding process has the following relationship:

[0119] ;

[0120] in, Denotes the joint loss function. Represents the total loss function of the QTRAN algorithm This represents the weights corresponding to the contrastive learning loss function.

[0121] Step 4: Sample small batches of data from the experience replay buffer, use the Adam optimizer to jointly update the parameters of the policy network and the value network to minimize the joint loss function, obtain the trained policy network, and deploy the trained policy network to each intersection agent to realize distributed signal control.

[0122] Please see Figure 2 Each agent receives local state information (including lane queuing status, current signal phase, traffic flow, and time-series traffic patterns), which is then used to extract and fuse spatial features through a graph convolutional network (GCN). The GCN can capture the spatial dependencies between intersections and extract traffic flow patterns and the mutual influences between intersections. In this way, the agent can not only obtain information from local states but also adjust its signal timing strategy according to the topology of the global traffic network, thereby making more accurate decisions.

[0123] After training, the agents at each intersection generate signal control actions in a distributed manner based on local real-time traffic conditions. The system supports an online learning mechanism, which collects new trajectory data and updates model parameters through continuous environmental interaction to achieve continuous optimization of control strategies. Each agent also shares learning experience with its neighboring nodes to collaboratively improve the overall control performance of the road network.

[0124] During model training, the system employs a centralized training and distributed execution framework. First, an experience replay buffer is initialized to store global experience replay samples. ,in, This represents the current global state (the set of observations from all intersections). For joint operations, As a global reward for environmental feedback, This represents the global state at the next time step. Each agent... i Local strategy network, individual action value network Central Joint Action Value Network and state value network parameters All parameters are randomly initialized and copied to the target network. During the interaction phase, each agent bases its actions on its local observations. (e.g., lane queuing, phase status, etc.) are obtained through an encoder to obtain a latent representation. The system outputs the action probability distribution through a local policy network and employs an ε-greedy policy to select actions: based on probability... Randomly select phase actions to explore the environment, with probability. Choose the optimal action for the current individual value network assessment. The actions of all agents constitute a joint action. a When applied to the SUMO simulation environment, the environment returns to the next state. With global rewards (Calculated by weighting indicators such as queue length and waiting time at each intersection), complete global experience replay sample It is stored in the buffer.

[0125] During the parameter update phase, each training step uniformly samples a mini-batch of empirical replay samples from the buffer. The following computation process is executed in parallel: First, the time difference target value for the next state is calculated using the target network, and then the time difference target value for each agent is obtained through forward propagation. Individual optimal action and combine into And then calculate ,in, The discount factor is used; secondly, the joint optimization objective consisting of four loss functions is calculated: the time difference loss function. Used to fit the Bellman equation; optimal action-constrained loss function loss function with non-optimal action constraints Jointly enforced joint value function With individual action value function Satisfying the QTRAN decomposition condition under arbitrary actions, achieving accurate credit allocation; contrastive learning loss function. Based on pseudo-labels generated by clustering, the loss function is used to narrow the distance between the potential representations of agents within the same functional region, thereby promoting policy differentiation learning.

[0126] The total loss function of QTRAN is a weighted sum:

[0127] ;

[0128] Subsequently, the joint loss function is calculated. ;Calculate the loss function using the Adam optimizer Regarding online parameters The gradient is calculated, and backpropagation is performed to update all network parameters. For stable training, the target network parameters are... Synchronize via soft update every C steps: ,in, To update the coefficients, through numerous iterations, the model effectively allocates global credits and coordinates regional strategies, while leveraging the enhanced representational capabilities of contrastive learning to ultimately learn a distributed signal control strategy adaptable to heterogeneous traffic flow patterns.

[0129] Please see Figure 4 , Figure 4 A schematic diagram of the simulated road network setup is provided. To verify the effectiveness of the proposed model, a 4×4 network-level road network was constructed on a traffic simulation simulator. This network consists of 16 signalized intersections. Each intersection has six lanes for each direction, with the east-west arterial road being a two-way four-lane design with a speed limit of 70 km / h; and the north-south road being a two-way two-lane design with a speed limit of 40 km / h. To construct a realistic traffic flow pattern, four main traffic flow paths were set up in the experiment: Path 1 (F1) includes traffic flow from E16 to E6 (…). Figure 4 (Indicated by the blue line) Path 2 (f1) contains traffic flow from E16 to E7 (light blue), Path 3 (f2) contains traffic flow from E16 to E8 (orange), and Path 4 (f2) contains traffic flow from E16 to E10 (light green). Fifteen minutes after the simulation begins, traffic flow on Paths 1 and 2 gradually decreases, while Paths 3 and 4 begin generating their corresponding traffic flows. These traffic flows include straight-ahead, left-turning, and right-turning traffic, and include some overlapping traffic flows to increase the challenge of the training task.

[0130] Please see Figure 5 , Figure 5This diagram illustrates the traffic flow generation for a simulated road network. In the experiment, increasing the traffic flow on different routes within the road network over time further tests the stability of the proposed model. The diagram shows the dynamic changes of these four types of traffic flow during the simulation period, including the generation, disappearance, and temporal evolution of traffic flow on each path. This design effectively simulates the spatiotemporal distribution characteristics of traffic demand in real urban road networks, providing a reliable testing environment for evaluating the performance of multi-agent cooperative control algorithms.

[0131] Please see Figure 6 , Figure 6 This is a schematic diagram of a four-phase intersection in a simulated road network. In this invention, for a conventional "cross-shaped" intersection with a left-turn lane in a synthetic road network, its selectable phase set is divided into four combined red-green signal phases (e.g., ...). Figure 6 The phases shown (a1, a2, ..., a4) represent the east-west straight and right turn phases (Phase I), east-west left and right turn phases (Phase IV), north-south straight and right turn phases (Phase II), and north-south left and right turn phases (Phase III). In a four-phase synthetic road network, each agent has the same action set, i.e., A = {a1, a2, a3, a4}. In complex real-world road networks, the number of signal phases and the phase sequence at different intersections are determined by the actual situation; the discrete phase switching command set {a1, a2, ..., ak} corresponds to a specific phase combination scheme for each action.

[0132] Please see Figure 7 , Figure 7 The training curves for four reinforcement learning-based vehicle-road cooperative control models are shown. To better observe the long-term trend of model performance, all methods were trained for 200,000 steps in the same 4×4 grid road network. The solid line in the figure represents the moving average of the average reward, and the shaded area near the solid line reflects the standard deviation range of the reward value. Overall, as the number of training steps increases, the agent gradually improves the policy quality through experience accumulation, resulting in a steady increase in the average reward value. Specifically, the MAPPO, MADDPG, and QTRAN models show relatively slow convergence speeds and fail to exhibit the same stability and convergence under the same number of training steps. In contrast, this invention shows rapid reward growth in the early stages of training and maintains a stable upward trend throughout the entire training period. This superior performance verifies the advantages of the proposed model in handling complex traffic flow scenarios.

[0133] Please see Figure 8 , Figure 8This section presents the average queue length curves for four multi-agent deep reinforcement models at intersections in a simulated road network. The average queue length is calculated by dividing the total number of queuing vehicles at all intersection entrances by the number of intersections. This indicator directly reflects the overall congestion level of the road network; therefore, this section selects average queue length as a key indicator for model evaluation. Figure 8 As shown in the figure, the reinforcement learning models QTRAN, MADDPG, MAPPO, and the results of this invention demonstrate that the algorithm within the framework proposed in this invention exhibits superior optimization control capabilities. As the simulation progresses, the average queue length of the other three reinforcement learning methods shows a continuous upward trend, indicating that their control strategies are ill-suited to dynamically changing traffic demands. In contrast, this invention maintains the lowest queue length level even as traffic volume increases, especially during high-load phases of the simulation. Other parameters are shown in Table 1.

[0134] Table 1: Evaluation Results of Other Indicators

[0135]

[0136] Regarding training efficiency, this embodiment compares the methods in completing the same number of training steps (1×10) in road networks of different sizes. 6 The training time required for this invention is roughly equivalent to MAPPO, but significantly lower than MADDPG and QTRAN. This efficiency advantage primarily stems from its hierarchical graph convolutional structure—by using cluster-based region partitioning, the global computational complexity is reduced from O(N²) to O(∑|C_k|²), allowing most graph convolutional operations to be executed in parallel within each cluster. Simultaneously, the unsupervised contrastive learning module reduces reliance on online interactive data through offline clustering and representation learning, further enhancing training efficiency.

[0137] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-agent traffic signal control method based on hierarchical contrastive learning, characterized in that, The method includes the following steps: Step 1: Construct a traffic simulation environment based on the traffic network, and model each intersection in the traffic simulation environment as an independent intelligent agent; The intelligent agent interacts with the SUMO simulation environment, collecting traffic state information, signal control actions, instant rewards, and traffic state information for the next state at each intersection, and storing them in the experience playback buffer. Step 2: Extract the latent representation of traffic state information using the policy network, group the latent representations of all agents using the K-means clustering algorithm to generate cluster labels, and use the cluster labels as pseudo-labels for different regions; divide all intersections into different functional regions according to the different regional pseudo-labels, construct a sub-graph for each functional region, and perform regional feature aggregation on the sub-graphs to obtain cluster-level state vectors. Step 3: Using different regional pseudo-labels as supervision signals, construct a contrastive learning loss function; using cluster-level state vectors as the abstract representation of traffic state in the corresponding functional areas, construct a QTRAN total loss function based on the QTRAN framework; and construct a joint loss function using the contrastive learning loss function and the QTRAN total loss function. Step 4: Sample small batches of data from the experience replay buffer, use the Adam optimizer to jointly update the parameters of the policy network and the value network to minimize the joint loss function, obtain the trained policy network, and deploy the trained policy network to each intersection agent to realize distributed signal control. In step 3, the contrastive learning loss function has the following relationship: ； in, Indicates batch size, This represents a positive sample with pseudo-labels in the same region. Let i and j represent the latent representations of the i-th and j-th agents, respectively. This indicates the cosine similarity calculation operation. Indicates samples within the same cluster; This represents a temperature parameter used to adjust the smoothness of the distribution. In step 3, the cluster-level state vector is used as the abstract representation of the traffic state of the corresponding functional area. Based on the QTRAN framework, the QTRAN total loss function is constructed, which specifically includes the following steps: potential representation Inputting a two-layer fully connected network to calculate the individual action value function of each agent at the intersection, the corresponding process has the following relationship: ； ； in, Indicates hidden features in the middle. This represents the trainable parameters of a fully connected network. , Represents the local state dimension; The function representing the value of an individual action of agent i; This represents the Q-value of all possible phase actions of agent i, where each action... The individual action value function is expressed as: ; Using cluster-level state vectors as abstract representations of traffic states in corresponding functional areas, K Cluster-level state vectors By fusing two independent fully connected networks, the joint action value function and state value function are calculated. The corresponding process has the following relationship: ； ； in, This represents the vectorized encoding of the joint action 'a', which consists of the individual actions of all agents. Let represent the weights and biases of the joint action value function, respectively. Let these represent the weights and biases of the state-value function, respectively. This represents vector concatenation. This represents the value function of the joint action. This represents the expected cumulative return under the global state and joint actions. Representing the global state and joint action respectively; Represents the state value function. This represents the expected return in the global state; Based on the QTRAN framework, the QTRAN total loss function is constructed using individual action value functions, joint action value functions, and state value functions.

2. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 1, characterized in that, In step 1, traffic state information is defined as phase state. Lane space occupancy Lane traffic flow Lane queue length ;in, This indicates the number of phases configured at the intersection. Indicates the number of import lanes. This indicates the number of spatial segments into which each lane is divided; Signal control actions are defined as the finite set of phases that an agent, consisting of all combinations of traffic signals from the traffic signal controller, can choose at an intersection; the set of selectable actions for each agent i. For this intersection, there is a finite set of phases; agent i starts from... Selecting and executing a phase action is called the individual action of the agent, denoted as . For a cross-shaped intersection with a left-turn lane in a composite road network, the selectable phase set is divided into four combined red-green signal phases. The four combined red-green signal phases are the east-west straight and right-turn phase, the east-west left-turn and right-turn phase, the north-south straight and right-turn phase, and the north-south left-turn and right-turn phase. The immediate reward is defined as the reward received by agent i from traffic environment feedback at discrete time step t. This traffic environment feedback includes local congestion relief, individual travel experience, and global supply-demand balance. The corresponding process has the following relationship: ； in, Indicates an immediate reward. This indicates the number of vehicles waiting behind the stop line in the m-th lane at the intersection entrance. Indicates vehicle v The current time step adds a waiting time. Indicates the pressure at the intersection. These represent the weighting coefficients corresponding to the number of vehicles waiting behind the stop line, the newly added waiting time at the current time step, and the intersection pressure, respectively. Indicates the static number of lanes. Represents the dynamic count of vehicles; Among them, vehicles j The calculation process for the new waiting time at the current time step follows the following formula: ； in, This indicates the moment when the system enters a waiting state. Indicates the index of the current time step. Indicates a historical time step. Indicates vehicle v In the historical time step The new waiting time, From 1 to An integer of -1.

3. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 2, characterized in that, In step 2, the potential representation of traffic state information is extracted, which specifically includes the following steps: Three independent encoders are set up, including a lane queuing encoder, a bird's-eye view encoder, and a time delay encoder; The lane queue length is input into the lane queue encoder. The lane queue length is first normalized to obtain... M Observation vector of each lane Then M The observation vectors of each lane are processed through a two-layer fully connected network to obtain lane queuing features. The lane queuing encoder uses a two-layer fully connected network, and the specific formula is as follows: ； ； in, This represents the output feature vector of the first fully connected layer. Represents the linear rectified activation function. Let represent the weight matrix and bias vector of the first fully connected layer, respectively. Let these represent the weight matrix and bias vector of the second fully connected layer, respectively. Indicates lane queuing characteristics, ; Lane space occupancy and lane traffic flow are input into the bird's-eye view encoder and reshaped into two three-dimensional tensors. The two three-dimensional tensors are then concatenated along the channel dimension to obtain the bird's-eye view feature map. ; Indicates the number of channels. The spatial dimensions are represented; then the bird's-eye view feature map is input into the ResNet-18 network, and the dimensions are adjusted through a linear layer to obtain the spatial topological features, as shown in the following formula: ； in, Representing spatial topological features, This represents the ResNet-18 deep residual network; Representing a linear layer involves performing global average pooling on the network output to obtain a vector representation, and then performing dimension mapping. The phase state is input to the time delay encoder. The current phase duration and period progress ratio are extracted from the phase state to obtain the time feature vector. Then, the time feature vector is mapped to a time embedding; the time delay encoder uses a multilayer perceptron, and the specific formula is as follows: ； in, Indicates time embedding, This represents a multilayer perceptron; By fusing lane queuing features, spatial topological features, and temporal embeddings through element-wise addition, a latent representation of traffic state information is obtained, and the corresponding formula has the following relationship: ； in, Representing a potential representation, This indicates an element-wise addition operation.

4. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 3, characterized in that, The specific steps for generating signal-controlled actions are as follows: The latent representation is input into the gated recurrent unit to capture temporal dependencies for time modeling. The corresponding process has the following relationship: ； ； ； ； in, This represents the Sigmoid activation function; This represents the hidden state at the previous time step, which is the state of the gated loop unit at time step [time step]. t Output -1; These represent the updated weight matrix and bias vector, respectively. This indicates updating the gate vector. This indicates resetting the gate vector. These represent the reset weight matrix and bias vector, respectively. Let these represent the weight matrix and bias vector used to calculate the candidate state, respectively. Represents the candidate hidden state vector. This indicates the hidden state after the current update. This indicates element-wise multiplication; The hidden state is represented by the output phase action probability distribution of a fully connected layer, and the corresponding process has the following relationship: ； in, This represents a policy function whose output is given by the traffic state. Next, select each phase action. The probability distribution; This represents the action performed by the agent at time step t. Indicates at time step t Traffic conditions observed by the intelligent agent Represents the normalized exponential function, These represent the trainable weight matrix and bias vector of the fully connected layer, respectively, representing the probability distribution of the generated actions. The phase action probability distribution is sampled to obtain the signal control action.

5. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 4, characterized in that, In step 2, the latent representations of all agents are grouped using the K-means clustering algorithm to generate cluster labels, which are then used as pseudo-labels for different regions. This process includes the following steps: potential representation After being processed by a gated recurrent unit, the features are used as input features for the clustering algorithm; where, Indicates the first N The potential representation of an agent , The dimension of the potential representation; With minimizing the sum of the within-cluster squared errors of all clusters as the optimization objective, the K-means clustering algorithm is applied to perform unsupervised clustering on N latent representations, minimizing the within-cluster squared errors of all clusters. The corresponding process has the following relationship: ； in, This indicates the preset number of clusters. This represents the latent representation of the i-th agent. Let K represent the sample set of the k-th cluster. Cluster The center vector; After clustering is completed, the final clustering result is obtained. Based on the final clustering result, a cluster label is assigned to each agent. The cluster label represents the number of the functional region to which it belongs. The cluster label is used as the region pseudo-label of the corresponding agent for subsequent policy differentiation learning and hierarchical credit allocation.

6. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 5, characterized in that, In step 2, based on different region pseudo-labels, all intersections are divided into different functional regions. A sub-graph is constructed for each functional region, and regional feature aggregation is performed on the sub-graphs to obtain cluster-level state vectors. Specifically, the steps include the following: Based on the pseudo-labels of the regions, all agents are assigned to the corresponding functional regions; Using intersection nodes within the current functional area as nodes, the actual road network topology connections or spatial proximity are defined as edges, and the Euclidean distance between intersections and road capacity are defined as edge weights, modeling this as a subgraph. , ;in, This represents the set of all intersection nodes belonging to this functional area. This represents the set of edges defined based on actual road network topology or spatial proximity. The calculation of edge weights follows the following formula: ； in, Indicates an intersection Intersection Edge weights between them Indicates an intersection Intersection The Euclidean distance between them Indicates road capacity. Indicates the scale parameter; The latent representation of each agent is used as the initial node feature of each subgraph node; Each subgraph is subjected to a two-layer graph convolutional network for feature propagation and aggregation to capture spatial dependencies within the region, thus obtaining region-aware features. The corresponding process has the following relationship: ； ； in, These represent the trainable weight matrices of the two layers of the GCN. Represents region-aware features. This represents the input feature matrix, where each row in the input feature matrix corresponds to a node in the subgraph. ; This represents an adjacency matrix with self-loops. , The adjacency matrix represents the subgraph. Represents the identity matrix. express The degree matrix; Max pooling is performed on all region-sensing features to generate the cluster-level state vector for that functional region. The corresponding process has the following relationship: ； in, Represents the cluster-level state vector. This represents the dimension of the cluster-level state vector.

7. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 6, characterized in that, The calculation of the QTRAN total loss function follows the following relationship: ； in, Represents the total loss function of QTRAN; Indicates the next global state; Represents global experience replay samples The reward item in the table represents the experience replay sample at discrete time steps. t Instant rewards generated ,Right now ; Parameters representing online networks; Represents the time difference loss function. This represents the optimal action constraint loss function. This represents the loss function for non-optimal action constraints. These represent the weights corresponding to the optimal action constraint loss function and the non-optimal action constraint loss function, respectively. The calculation process of the time difference loss function is based on the following relationship: ； ； in, This indicates the combined action value of online network output; Indicates the discount factor; This represents the optimal combination of actions for the individual in the next state. This represents the optimal combination of individual actions chosen by the target network in the next global state, based on its policy. The estimated value; These represent the parameters of the target network. Indicates the target value; The calculation process of the optimal action constraint loss function is based on the following relationship: ； in, Represents the state-value function, outputting the state. The baseline value; This represents a fixed estimate of the sum of individual action values, i.e., a fixed value used when calculating the optimal action constraint loss function. Not participating in gradient update, defined as ; This represents the optimal action of agent i. Q value, The optimal action of agent i is calculated by the target network. This represents the optimal action combination for the current individual; , Let N be the individual optimal action of agent N, where N is the optimal action of agent N. ; Non-optimal action constraint loss function: ensures that for any non-optimal joint action... a The combined value of their actions shall not exceed the sum of their individual values ​​minus the baseline. ； in, B This represents a set of non-optimal joint actions sampled from experience; Represents a linear rectified function; Indicates the corresponding action a A fixed estimate of the sum of individual values, i.e., a fixed value when calculating the non-optimal action constraint loss function. It does not participate in gradient updates; , Represents the action of agent i The Q value is calculated from the target network.

8. The multi-agent traffic signal control method based on hierarchical contrastive learning according to claim 7, characterized in that, In step 3, a joint loss function is constructed using the contrastive learning loss function and the QTRAN total loss function. The corresponding process has the following relationship: ； in, Denotes the joint loss function. This represents the total loss function of the QTRAN algorithm. This represents the weights corresponding to the contrastive learning loss function.

Images