Generalizable graph multi-agent reinforcement learning signal control method and system for heterogeneous intersections

Abstract

The application discloses a heterogeneous intersection-oriented generalizable graph multi-agent reinforcement learning signal control method and system, and belongs to the technical field of intelligent driving and intelligent traffic, comprising the following steps: S1, abstracting a road network into a graph, with nodes corresponding to heterogeneous intersections, and taking static attributes of each intersection as a node identity identifier; S2, modeling multi-intersection signal cooperative control as a distributed partially observable Markov decision process, extracting node spatiotemporal features by using a diffusion convolution recurrent network, enhancing node representation by means of an attribute-guided heterogeneous diffusion attention mechanism, outputting signal light phase instructions by a shared Actor network, and training by adopting a centralized training-distributed execution framework; and S3, based on a current signal light phase and a remaining time thereof, calculating a smooth target speed for a target intelligent networked vehicle in real time and issuing the smooth target speed for execution. The application realizes explicit modeling of heterogeneous intersections and zero-sample cross-road network migration, reduces migration cost, and improves traffic efficiency and system stability.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving and intelligent transportation technology, and in particular relates to a generalizable graph multi-agent reinforcement learning signal control method and system for heterogeneous intersections. Background Technology

[0002] Traffic signal control systems are a core component of urban traffic management, and the quality of their control strategies directly impacts road network efficiency, vehicle delays, energy consumption, and environmental quality. With accelerated urbanization and a continuous increase in motor vehicle ownership, the contradiction between urban traffic supply and demand is becoming increasingly prominent, and road network congestion is showing a trend towards normalization and regionalization. This places higher demands on the adaptive capabilities, collaborative capabilities, and intelligence level of signal control systems.

[0003] Existing traffic signal control methods mainly include rule-based, optimization-based, and learning-based methods. Rule-based methods are the most traditional and widely used, with their decision-making logic entirely determined by manually preset rules. For example, timed control sets fixed timing schemes based on historical traffic data and operates on a fixed cycle. It is simple to implement and easy to manage, but it cannot detect real-time changes in traffic conditions. When the actual traffic flow deviates from the preset scheme, phenomena such as green lights being left idle or queue overflows may occur. Inductive control, on the other hand, uses cross-sectional detectors deployed in front of the stop line to detect vehicle arrivals in real time and dynamically extend or shorten the green light time of the current phase. It achieves a certain degree of response to random traffic flow, but it lacks global coordination capabilities and is prone to regional coordination failures when traffic demand is close to saturation or when there is strong coupling between road networks. Optimization-based methods transform signal timing problems into mathematical programming or combinatorial optimization problems, using linear programming, dynamic programming, or metaheuristic algorithms to find the optimal values ​​of decision variables. Theoretically, these methods can find the optimal solution under given constraints and have good mathematical interpretability. However, as the road network scale expands and the number of optimization variables increases, the solution complexity rises sharply, making it difficult to meet the time constraints of real-time control. At the same time, the structure of the optimization model is directly coupled with the road network topology, requiring remodeling and solving when using road networks of different sizes.

[0004] Learning-based methods, particularly deep reinforcement learning, have garnered significant attention in recent years. These methods model signal control as a Markov decision problem, where the agent learns the optimal timing strategy through continuous interaction and trial and error with the traffic environment, aiming to maximize cumulative rewards. This allows them to perceive complex traffic conditions and adaptively adjust decisions. However, most existing deep reinforcement learning methods are designed for fixed road networks. Their model structure is strongly coupled with the number of nodes and connections within a specific road network. When applied to new road networks with different structures, they cannot be directly loaded due to input dimension mismatch, requiring redesign of the state and action spaces and retraining from scratch, resulting in extremely high deployment costs. Furthermore, these methods typically treat intersections as homogeneous nodes, failing to explicitly model the differences in control logic between intersections with different topologies. In addition, some research attempts to leverage large language models for traffic scene understanding and policy generation, but these large models lack the ability to accurately model the spatiotemporal dynamics of traffic flow and the physical topology of the road network, suffer from high inference latency, and lack unified representation methods for heterogeneous observations, making cross-road network adaptation difficult.

[0005] In summary, existing traffic signal control methods generally suffer from the following shortcomings: they lack the ability to explicitly model intersection heterogeneity, failing to distinguish the differences in control logic between intersections with different topologies; and their model structures are strongly coupled to specific road networks, lacking zero-shot generalization transfer capabilities across road networks. Model transfer requires high retraining costs, severely hindering large-scale application. Therefore, there is an urgent need for a general signal cooperative control method that can explicitly model intersection heterogeneity and possess zero-shot transfer capabilities across road networks. Summary of the Invention

[0006] This invention aims to address the following issues: existing traffic signal control methods lack the ability to explicitly model the heterogeneity of intersections, cannot distinguish the differences in control logic between intersections with different topologies, and have a model structure that is strongly coupled with a specific road network, lacking the ability to generalize and transfer data across road networks with zero samples, and requiring high retraining costs for model transfer.

[0007] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections, comprising the following steps: S1. Abstract the road network into a graph, where nodes correspond to heterogeneous intersections and edges correspond to directional roads connecting the intersections. Extract the static attributes of each intersection as the identity identifier of the corresponding node. S2. Based on the above figure, the multi-intersection signal cooperative control problem is modeled as a distributed partially observable Markov decision process, in which each signal-controlled intersection is an independent agent, each agent obtains its own local observations, and achieves cooperative optimization by sharing global rewards; Spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network, and the dynamic hidden state of each intersection at the current moment is output. The static attributes of each node are mapped to node identity embeddings through an attribute-guided heterogeneous diffusing attention mechanism. The dynamic hidden states of neighboring nodes are then aggregated with attention weighting based on the node identity embeddings to obtain the enhanced node representation. Each agent takes its own local observations, the enhanced node representations, and the static attributes as inputs, and outputs the signal light phase command for the next stage of the intersection through the shared Actor network; The shared Actor network and the global Critic network are trained using a centralized training-distributed execution framework. During the training phase, the global Critic network estimates the global value based on the dynamic hidden states of all nodes, and uses the trajectories collected by each agent to jointly update the parameters of the shared Actor network and the global Critic network. During the execution phase, each agent independently outputs the traffic light phase command through the shared Actor network, and the global Critic network is not enabled.

[0008] Furthermore, in S1, the static attribute is a five-dimensional vector φ_i determined by the following formula:

[0009] in, This indicates the number of lane turning connections currently controlled by the intersection's traffic lights. and These represent the total number of inbound lanes and the total number of outbound lanes, respectively. Indicates the number of active phases at the current intersection. This indicates the number of lanes contained in the current green light phase p. To prevent extremely small positive numbers with a denominator of zero, , , These are the corresponding normalization constants.

[0010] Furthermore, in S2, the enhanced node representation obtained through the attribute-guided heterogeneous diffusion attention mechanism specifically includes: The static attributes are mapped to node identities using a multilayer perceptron and embedded as keys. The query value is obtained by linearly projecting the dynamic hidden state. ; Calculate the heterogeneous similarity matrix S between nodes using scaled dot product attention: ,in, The dimension of the attention head; Introducing a mask matrix When node i is adjacent to node j or i=j ,otherwise The mask matrix is ​​applied to the heterogeneous similarity matrix and normalized to obtain the attention weights. ; The dynamic hidden state of neighboring nodes is determined based on the attention weights. Weighted aggregation is performed to obtain enhanced heterogeneous spatiotemporal features. : , where N(i) is the set of neighboring nodes of node i.

[0011] Furthermore, during the training phase, the parameters of the shared Actor network and the global Critic network are jointly updated using the trajectories collected by each agent through a proximal policy optimization algorithm. The hybrid reward during the intensive training phase is a weighted sum of the agent's local reward and the global average reward, calculated as follows:

[0012] in, Let α be the local reward of agent i at time t, N be the total number of agents, and α be the weight coefficient.

[0013] Furthermore, the method also includes S3: The system receives the current traffic light phase and its remaining time corresponding to the traffic light phase command, obtains the real-time number of vehicles in the queue, the physical length of the queue, the distance to the vehicle in front, and the speed of the vehicle in front in the lane where the target intelligent connected vehicle is currently located, and calculates and smooths the target speed of the target intelligent connected vehicle in real time through space-time coupled target planning and issues the plan for execution.

[0014] Furthermore, when deploying to the target road network after training is completed, the trainable weights of the shared Actor network are frozen, and the graph structure is dynamically reconstructed through the adjacency matrix reset interface according to the actual number of nodes in the target road network, and the corresponding action mask is regenerated. Each agent takes its local observations, the enhanced node representations, and the static attributes as inputs, and infers the action probability distribution through the shared Actor network. After sampling, it outputs traffic light phase commands and issues them for execution. The global Critic network is not enabled during the deployment phase to achieve zero-sample transfer to the target road network without retraining.

[0015] Secondly, the present invention provides a generalizable graph multi-agent reinforcement learning signal control system for heterogeneous intersections, comprising: The graph construction module is used to abstract the road network into a graph, where nodes correspond to heterogeneous intersections, edges correspond to directional roads connecting the intersections, and the static attributes of each intersection are extracted as the identity identifiers of the corresponding nodes. The signal decision module is used to model the multi-intersection signal cooperative control problem as a distributed partially observable Markov decision process based on the above diagram, where each signal-controlled intersection is an independent agent and each agent acquires its own local observations. Spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network, and the dynamic hidden state of each intersection at the current moment is output. The static attributes of each node are mapped to node identity embeddings through an attribute-guided heterogeneous diffusing attention mechanism. The dynamic hidden states of neighboring nodes are then aggregated with attention weighting based on the node identity embeddings to obtain the enhanced node representation. Each agent takes its own local observations, the enhanced node representations, and the static attributes as inputs, and outputs the signal light phase command for the next stage of the intersection through the shared Actor network; The shared Actor network and the global Critic network are trained using a centralized training-distributed execution framework; The trajectory smoothing module receives the current traffic light phase and its remaining time, and calculates and smooths the target speed of the intelligent connected vehicle in real time through space-time coupled target planning and sends it out for execution.

[0016] Thirdly, the present invention provides an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in the first aspect.

[0017] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in the first aspect.

[0018] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: To address the lack of explicit modeling capabilities for intersection heterogeneity in existing technologies, this invention uses the static attributes of each intersection as identifiers for graph nodes. Through an attribute-guided heterogeneous diffusion attention mechanism, the node identities encoded by static attributes are embedded to guide feature aggregation between nodes. This enables the model to distinguish the differences in control logic between intersections with different topologies, effectively capturing the spatiotemporal coupling relationship of traffic flow and improving the model's adaptability to changes in road topology. To address the issues of strong coupling between model structure and specific road networks, and the high retraining costs required for cross-road network migration, this invention, based on static attribute identification and dimension-unified observation design, enables trained models to be deployed to target road networks of different sizes and phase configurations with zero samples without retraining. This is achieved through adjacency matrix resetting and action mask regeneration. This overcomes the performance degradation problem caused by structural differences during cross-road network migration in traditional methods and reduces the model retraining costs in practical applications. This invention achieves multi-agent collaborative optimization through a centralized training-distributed execution framework, combined with a lower-level control strategy for smoothing the trajectory of intelligent connected vehicles. While improving the overall traffic efficiency of the road network, reducing average waiting time and queue length, it effectively suppresses traffic flow oscillations and enhances system stability. It provides an end-to-end solution for the intelligent management and control of urban traffic signals that is efficient, generalizable, and feasible. Attached Figure Description

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will now be described in detail with reference to the accompanying drawings, wherein... Figure 1 The flowchart shows the generalizable graph multi-agent reinforcement learning signal control method and system for heterogeneous intersections according to the present invention. Figure 2 This is a heterogeneous road network scenario diagram according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the training process of the centralized training-distributed execution framework of this invention. Figure 4 This is a reward convergence curve during the training phase of this invention. Figure 5 This is the generalized spatiotemporal trajectory diagram of the present invention under a 2×2 heterogeneous training road network; Figure 6 This is the generalized spatiotemporal trajectory diagram of the present invention under a 4×4 isomorphic road network; Figure 7 This is the generalized spatiotemporal trajectory diagram of the present invention in a real heterogeneous road network with 14 intersections. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures, and should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0023] Example 1 This embodiment provides a generalizable graph multi-agent reinforcement learning signal control method and system for heterogeneous intersections, such as... Figure 1 As shown, this method is implemented in a vehicle-road cooperative system consisting of a signal controller, roadside sensing devices, roadside units (RSUs), and intelligent connected vehicles. The signal controller acts as the upper-level decision-making carrier, executing road network map construction and traffic light phase decisions at each intersection. The roadside sensing devices provide the signal controller with traffic state perception data for each controlled lane. The roadside units issue speed guidance commands to the lower-level intelligent connected vehicles. In terms of deployment, this method can be executed locally by the signal controllers at each intersection, or it can be executed by the edge computing server of the regional traffic control center after unified decision-making for multiple intersections.

[0024] This method abstracts the urban road network into a topological graph, treating each intersection as a graph node and road connections as edges. At the upper layer, it models traffic signal cooperative control as a multi-agent decision-making problem on the graph. First, it extracts the static structural attributes of each intersection to construct a heterogeneous fingerprint. Then, it uses a diffusing convolutional recurrent network to model the spatiotemporal dependencies of traffic flow and introduces an attribute-guided heterogeneous diffusing attention mechanism to enhance the generalization ability of the graph representation. Finally, it employs a proximal policy optimization algorithm within a centralized training-distributed execution framework to achieve multi-agent cooperative decision-making. Furthermore, at the lower layer, it combines time-space planning to develop a trajectory smoothing strategy for vehicle generation.

[0025] This method effectively overcomes the performance degradation problem caused by the difference in intersection structure when traditional models migrate to heterogeneous road networks. It can achieve zero-sample generalization deployment to road networks of different sizes and topologies without retraining. At the same time, through the joint optimization of upper-layer signal coordination and lower-layer vehicle smoothing, it significantly improves the overall traffic efficiency and traffic flow stability of the road network, providing an end-to-end solution with both generalization ability and practical value for the intelligent management and control of large-scale urban traffic signals.

[0026] The overall approach comprises three stages: abstracting the road network into a graph, upper-level macroscopic signal phase decision-making, and lower-level microscopic vehicle trajectory smoothing. The specific implementation steps are as follows: (1) Abstract the road network into a graph S1. Abstract the road network as a graph, where nodes correspond to heterogeneous intersections, edges correspond to directional roads connecting the intersections, and each node's identity is identified by the static attributes of each intersection; specifically including: S11. Abstract each signal-controlled intersection in the road network into a graph node, extract the lane-level connection relationship between each intersection, and construct a set of directed edges between nodes. The road network in this embodiment is a road network extracted from the real environment and simplified to a certain extent. In other embodiments, the road network can be a controllable road network of any size and topology.

[0027] S12. Extract the static attributes of each intersection as the identity identifier of the corresponding graph node; in this embodiment, the static attributes adopt the following five-dimensional vector. In other embodiments, the static attributes may also include static features that can identify the differences in intersection structure and control, such as intersection geometry type and signal control type. The static attributes include the following state variables: , in, This indicates the number of lane turning connections currently controlled by the intersection's traffic lights. and These are the total number of import lanes and export lanes, respectively. This is the number of valid phases at the current intersection. This represents the number of lanes in the current green light phase, where ε is a very small positive number to prevent the denominator from being 0. , , These are the corresponding normalization constants.

[0028] (2) Phase decision of upper-level macroscopic signals S2. Based on this diagram structure, the multi-intersection signal coordination control problem is modeled as a distributed partially observable Markov decision process. The spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network in the feature layer. An attribute-guided heterogeneous diffusing attention mechanism is introduced to enhance the node representation. Finally, the next stage signal light phase command for each intersection is output through the policy layer. Specifically, the following steps are included: S21. The multi-intersection signal cooperative control problem is modeled as a distributed partially observable Markov decision process, where each signal-controlled intersection is treated as an independent agent. Each agent only acquires its own local observations and achieves cooperative optimization by sharing a global reward. The observation space, action space, and reward function of each agent are as follows: The observation space uses phases as the basic unit, forming a fixed-dimensional phase feature vector by aggregating the traffic states of all controlled lanes under each phase. Since different intersections have different numbers of lanes and phases, the upper limit of the number of phases for all intersections is uniformly set to the global maximum value P_max. Missing phases are filled with zero vectors, ensuring that the observation tensor dimension is completely consistent for any intersection. For any intersection, its observation vector is composed of three concatenated parts: 1) The eigenvector of each phase is represented by the following quintuple: , in, Indicates the current green light phase. The normalized queue ratio is defined as the proportion of vehicles with speeds below 0.1 m / s in all lanes controlled by that phase. It is the normalized density, which is the ratio of the current number of vehicles to the maximum number of vehicles in the saturated time slot. It is the normalized cumulative waiting time. It is the upstream inflow intensity, defined as the normalized value of the number of vehicles that are more than 30 meters away from the intersection and traveling at a speed of more than 3 m / s in the entrance lane of that phase; 2) The duration of the green light has been normalized to This means knowing how long the current green light has lasted, which helps avoid traveling for only one green light phase. 3) The one-hot encoding of the action in the previous decision step is denoted as... .

[0029] The action space refers to the space at each decision point (every 15 seconds). At each decision point, the agent selects a phase from its own set of green light phases A_i as the phase for the next stage of operation; the output is the index of the selected phase. In this embodiment, the decision interval is 15 seconds. In other embodiments, the decision interval can be set according to actual control requirements.

[0030] , The reward function includes queuing penalties. Maximum waiting time penalty Parking penalties Phase switching penalty This allows intelligent agents to learn traffic flow patterns from different directions, thereby improving traffic efficiency. , in, Queueing penalties are calculated by counting the number of vehicles traveling at speeds below 0.1 m / s in all approach lanes of the intersection. Summation and normalization are used to guide agents to prioritize the release of phases with longer queues, avoiding network deadlock caused by queue backtracking. , The maximum waiting time penalty is calculated by taking into account the waiting time of all vehicles in the controlled lanes at the intersection. The maximum value is normalized to prevent individual vehicles from waiting too long due to a prolonged green light in one direction: , Parking penalty statistics show the number of vehicles newly entering the parking state within the current decision interval. And normalization is used to encourage agents to reduce unnecessary braking and stopping, thereby improving traffic flow smoothness and efficiency: , Phase switching penalty applies a fixed penalty when the agent selects a phase to switch, which is used to suppress frequent and meaningless phase transitions and ensure traffic safety. .

[0031] S22. DCRNN is used to extract the spatiotemporal features of each node. Diffusion convolution simulates the random walk of traffic flow on the graph to aggregate the spatial neighborhood influence, and gated recurrent units model the temporal sequence. Finally, the dynamic hidden state of each intersection at the current time is output. As a graph-aware context representation of the policy network, the network structure parameters such as the number of diffusion steps, hidden dimension, and attention head dimension of the diffusion convolution can be adjusted according to computing resources and road network scale. The specific values ​​given in Example 2 are preferred implementations.

[0032] S23. Introducing AHDA to enhance node representation by encoding static node attributes, thereby improving the model's generalization ability when transferring across road networks; specifically including: A multilayer perceptron is used to map static attributes to node identities and embed them as keys. Simultaneously, the query value is obtained by linearly projecting the hidden state. Then, the heterogeneous similarity matrix S between nodes is calculated using scaled dot product attention: , in, For the dimension of attention head.

[0033] To conform to the physical topological constraints of the transportation network, a mask matrix is ​​introduced. When node i is adjacent to node j or i=j Otherwise The mask is applied to the similarity matrix and normalized to obtain the final attention weights. : .

[0034] Then, the hidden states of neighboring nodes are analyzed using this attention weight. The final enhanced heterogeneous spatiotemporal features are obtained by reweighting and aggregating. : .

[0035] S24. A centralized training-distributed execution framework is adopted. During execution, each agent takes local observations, AHDA context, and static attributes as input, outputs phase commands and collects trajectories through a shared Actor network. During training, the global Critic estimates the global value based on the hidden states of all nodes and updates the network parameters using the collected trajectories through a proximal policy optimization algorithm. This embodiment uses the proximal policy optimization algorithm (PPO). In other embodiments, other reinforcement learning algorithms based on policy gradients can also be used to update the network parameters. The training phase includes: During the intensive training phase, to balance local and global objectives, a hybrid reward is defined as a portion of the agent's local reward r_i,t and a portion of the global average reward, with the weights controlled by α, calculated as follows: .

[0036] (3) Smooth vehicle trajectories in the lower microscopic layer S3. Receive the current traffic light phase and its remaining time from the upper layer, and calculate and smooth the target speed for the target intelligent connected vehicle in real time through space-time coupled target planning, then issue the smoothed target speed for execution. The smoothed target speed is calculated by the signal controller or roadside unit, and sent to the target intelligent connected vehicle via the vehicle-road cooperative communication link, where it is executed by the onboard control unit. Specifically, this includes the following steps: S31. Receive the current traffic light phase and its remaining time from the upper layer, obtain the real-time queue number of vehicles in the lane currently occupied by the target vehicle, the physical length of the queue, the distance to the vehicle in front, and the speed of the vehicle in front, and perform spatial and temporal planning; wherein the spatial and temporal planning are as follows: Spatial planning distance This involves calculating the dynamic distance between the current vehicle and the rear of the queue of vehicles ahead: , in, The distance between the vehicle and the stop line at the intersection. For the number of cars in the queue, To create effective queuing space for bicycles, To reserve a safety margin.

[0037] Planning Time To plan based on the upper-level signal intent and the current number of cars in the queue, assume... To determine the saturation headway, if the current light is green ( =0) means only the time required for the queue of vehicles ahead to clear is considered, if it is a red light ( =1) Then we need to take into account the sum of the waiting time for the next green light and the time required for the queue of vehicles ahead to clear: .

[0038] S32. After target planning, the target speed of the intelligent connected driving vehicle is calculated and smoothed in real time and then executed; the target speed is calculated as follows: The target speed is obtained by combining the distance from spatial planning with the remaining time from time planning. .

[0039] In this embodiment and In other embodiments, 2 m / s and 13.89 m / s were used respectively. and It can be set according to road speed limits and actual safety needs.

[0040] Example 2 This embodiment is based on the following assumptions: (1) In this embodiment, some vehicles in the road network are intelligent connected driving vehicles with V2X communication capabilities, which can receive and execute speed guidance instructions issued by the roadside unit; the upper-level signal controller and the lower-level intelligent connected driving vehicles interact in real time through the vehicle-road cooperative communication link, and the signal phase status and queue length information are broadcast. The queue dissipation status formed after the intelligent connected driving vehicle responds and executes is transmitted back to the signal controller through the roadside sensing device, forming a complete closed loop.

[0041] (2) The lower-level speed control has built-in safety redundancy constraints.

[0042] First, the speed guidance command is constrained by the dual boundary constraints of the maximum road speed limit of 13.89 m / s and the minimum rolling speed of 2 m / s, to prevent the vehicle from coming to a complete stop or speeding. Secondly, a headway safety check is introduced. When the distance between the intelligent connected vehicle and the vehicle in front or the stop line is less than the preset safety threshold of 2m, the system automatically hands over vehicle control to the SUMO underlying Krauss following model, which executes the emergency braking collision avoidance logic to ensure that no collision occurs under any conditions.

[0043] In other embodiments, the underlying car-following model is not limited to the Krauss model of SUMO, but can also be a car-following model with emergency braking and collision avoidance capabilities, such as IDM.

[0044] like Figure 1 As shown, this paper presents a generalizable graph multi-agent reinforcement learning signal control method and system for heterogeneous intersections. The specific content of the method is as follows: (1) Abstract the road network into a graph S1. Abstract the road network as a graph, where nodes correspond to heterogeneous intersections, edges correspond to directional roads connecting intersections, and each node's identity is identified by the static attributes of each intersection.

[0045] S11 abstracts each signal-controlled intersection in the road network as a graph node, extracts the lane-level connection relationship between each intersection, and constructs a set of directed edges between nodes; S12 extracts the static attributes of each intersection as the identity identifier of the corresponding graph node; In S11, the road network is a heterogeneous road network extracted from the real environment and simplified to a certain extent. This road network contains a four-way... The road network structure is as follows: An intersection with three lanes in each direction, a two-lane intersection in each direction, and a three-way mixed lane and a four-way mixed lane. Figure 2 As shown; In S12, the static attributes include the following state variables: (1) in, This indicates the number of lane turning connections currently controlled by the intersection's traffic lights. and These are the total number of import lanes and export lanes, respectively. This is the number of valid phases at the current intersection. This represents the number of lanes in the current green light phase, with the first three values ​​ranging from [0,3], the fourth value ranging from [0,2], and the last value ranging from [0,1].

[0046] (2) Phase decision of upper-level macroscopic signals S2. Based on this diagram structure, the multi-intersection signal cooperative control problem is modeled as a distributed partially observable Markov decision process. The spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network in the feature layer. An attribute-guided heterogeneous diffusing attention mechanism is introduced to enhance the node representation. Finally, the signal light phase command for the next stage of each intersection is output through the policy layer.

[0047] S21. The problem of coordinated control of signal signals at multiple intersections is modeled as a distributed partially observable Markov decision process, in which each signal-controlled intersection is an independent agent, each agent only obtains its own local observations, and achieves coordinated optimization by sharing global rewards. S22. The DCRNN is used to extract the spatiotemporal features of each node. Spatially, an adjacency matrix of the road network topology is constructed, and diffusing convolution is used to simulate the bidirectional random walk process of traffic flow on the graph structure. Within a finite number of diffusion steps, the spatial influence of neighboring nodes is aggregated to capture the propagation pattern of traffic flow between different intersections. Temporally, gated recurrent units are used to perform temporal state transfer and update of multi-step historical observations, modeling the continuous evolution process of traffic state. Finally, the dynamic hidden state h_j of each intersection at the current decision time is output as the graph-aware context representation for subsequent reinforcement learning strategies. S23. Introduce AHDA to enhance node representation by encoding static node attributes, thereby improving the generalization ability of the model when transferring across road networks. S24. A centralized training-distributed execution framework is adopted. During the execution phase, each agent takes local observations, context extracted by AHDA, and static attributes as inputs, outputs the action probability distribution through the shared Actor network, and outputs the traffic light phase instructions for the next stage of each intersection after action masking. At the same time, the agent continuously interacts with the environment, storing the experience such as state, action, and reward for each step into a buffer. During the training phase, multi-step trajectories are sampled from the buffer. The global Critic network receives the hidden states of all nodes after DCRNN encoding, outputs the global state value after mean pooling, and calculates the generalized advantage estimate based on this. Finally, a proximal policy optimization algorithm is adopted to jointly update the parameters of the shared Actor network and the global Critic network by maximizing the policy ratio target after pruning and minimizing the value loss. In S21, the observation space, action space, and reward function of each agent are as follows: The observation space uses phase as the basic unit, forming a fixed-dimensional phase feature vector by aggregating the traffic states of all controlled lanes under each phase. Since different intersections have different numbers of lanes and phases, the upper limit of the number of phases for all intersections is uniformly set to a global maximum of 8. Missing phases are filled with zero vectors, ensuring that the observation tensor dimension is completely consistent for any intersection. For any intersection, its observation vector is composed of three concatenated parts: 1) The eigenvector of each phase is represented by the following quintuple: (2) in, This indicates the current green light phase; it is 1 if the light is green and 0 otherwise. The average queuing rate of the lanes controlled by this phase is defined as the proportion of vehicles with a speed of less than 0.1 m / s in all lanes controlled by this phase, and its value ranges from [0,1]. It is the average density of the lane controlled by this phase, defined as the ratio of the current number of vehicles to the maximum number of vehicles under the saturation time slot, and its value range is [0,1]. It is the average cumulative waiting time of the lane controlled by this phase, and its value is normalized to [0,1]. It is the upstream vehicle index of the lane controlled by this phase, defined as the total number of vehicles that are more than 30 meters away from the intersection and traveling at a speed of more than 3 m / s in the entrance lane of this phase, and then normalized to [0,1]. 2) The duration of the green light has been normalized to This means knowing how long the current green light has lasted, which helps avoid traveling for only one green light phase. 3) The one-hot encoding of the action in the previous decision step is denoted as... That is, only one phase was selected out of the eight phases in the previous time step, which is 1.

[0048] The action space is the set of green light phases that the agent needs to select from its own at each decision moment (15-second interval). Select one phase as the running phase for the next stage; that is, the output is the index of the selected phase.

[0049] (3) The reward function includes queuing penalties. Maximum waiting time penalty Parking penalties Phase switching penalty This allows intelligent agents to learn traffic flow patterns from different directions, thereby improving traffic efficiency. (4) The queuing penalty is calculated by counting the number of vehicles traveling at speeds below 0.1 m / s in all approach lanes of the intersection. Summation and normalization are used to guide agents to prioritize the release of phases with longer queues, avoiding network deadlock caused by queue backtracking. (5) The maximum waiting time penalty is calculated by taking into account the waiting time of all vehicles in the controlled lanes at the intersection. The maximum value is normalized to prevent individual vehicles from waiting too long due to a prolonged green light in one direction: (6) Parking penalty statistics show the number of vehicles newly entering the parking state within the current decision interval. And normalization is used to encourage agents to reduce unnecessary braking and stopping, thereby improving traffic flow smoothness and efficiency: (7) Phase switching penalty applies a fixed penalty when the agent selects a phase to switch, which is used to suppress frequent and meaningless phase transitions and ensure traffic safety. (8) In S23, the AHDA implementation logic specifically includes: A multilayer perceptron is used to map static attributes to node identities and embed them as keys. Simultaneously, the query value is obtained by linearly projecting the hidden state. Then, scaling dot product attention is used to calculate the heterogeneous similarity matrix between nodes. : (9) in, For the dimension of attention head.

[0050] To conform to the physical topological constraints of the transportation network, a mask matrix is ​​introduced. When node i is adjacent to node j or i=j Otherwise The mask is applied to the similarity matrix and normalized to obtain the final attention weights. : (10) Then, the hidden states of neighboring nodes are analyzed using this attention weight. The final enhanced heterogeneous spatiotemporal features are obtained by reweighting and aggregating. : (11) In S24, the training phase reward is calculated as follows (see details). Figure 3 ): In the intensive training phase, to balance local and overall objectives, the hybrid reward is defined as the agent's local reward. Each component, along with the global average reward, comprises a portion, with the weight controlled by α, calculated as follows: (12) In S24, the hyperparameter settings during training are shown in Table 1: Table 1 Training Hyperparameter Settings

[0051] In the distributed execution phase, the Actor network, where all agents share the same set of network parameters, consists of a two-layer feature extraction backbone and a one-layer residual branch. The backbone accepts a concatenated input of 49-dimensional local observations, 64-dimensional graph context, and 5-dimensional static attributes (a total of 118 dimensions). After passing through a 256-dimensional fully connected layer, layer normalization, and Dropout, it merges with the feature branch mapped by a 128-dimensional fully connected layer. Finally, the output layer maps and masks the corresponding action probability distribution. The global Critic network performs mean pooling on the 64-dimensional hidden states of all nodes, then maps them sequentially through 256-dimensional and 128-dimensional fully connected layers to a 1-dimensional state value. The graph encoder uses a single-layer diffusing convolutional gated recurrent unit with 3 diffusing steps and 64 hidden dimensions, stacked with an AHDA attention module. The latter encodes 5-dimensional static attributes into a 64-dimensional key vector, calculates heterogeneous interaction weights between nodes through a masked attention mechanism, and then outputs an enhanced 64-dimensional node context representation after passing through a diffusing convolutional layer. The reward convergence curve after training is shown in the figure. Figure 4 As shown.

[0052] In this embodiment, the training is completed in the offline SUMO traffic simulation platform. The shared Actor network parameters obtained from the training are then fixed and deployed to the signal controller or edge computing server. In other embodiments, other platforms that support road network traffic flow simulation can also be used to complete the training.

[0053] (3) Smooth vehicle trajectories in the lower microscopic layer S3 receives the current traffic light phase and its remaining time from the upper layer, calculates and smooths the target speed of the target intelligent connected vehicle in real time through space-time coupled target planning, and sends it down for execution.

[0054] S31. Receive the current traffic light phase and its remaining time output from the upper layer, obtain the real-time number of vehicles in the queue, physical length of the queue, distance to the vehicle in front, and speed of the vehicle in front in the lane where the target vehicle is currently located, and perform spatial planning and time planning. S32. After target planning, calculate and smooth the target speed of the intelligent connected driving vehicle in real time and issue the execution order. In S31, spatial programming and time programming calculations are performed as follows: Spatial planning distance This involves calculating the dynamic distance between the current vehicle and the rear of the queue of vehicles ahead: (13) in, The distance between the vehicle and the stop line at the intersection. For the number of cars in the queue, To create effective queuing space for bicycles, To reserve a safety margin.

[0055] Planning Time To plan based on the upper-level signal intent and the current number of cars in the queue, assume... To determine the saturation headway, if the current light is green ( =0) means only the time required for the queue of vehicles ahead to clear is considered, if it is a red light ( =1) Then we need to take into account the sum of the waiting time for the next green light and the time required for the queue of vehicles ahead to clear: (14) In S32, the target velocity is calculated as follows: The target speed is obtained by combining the distance from spatial planning with the remaining time from time planning. (15) After training, testing is performed by loading the trained network parameters, freezing all trainable weights, and not updating any parameters. Based on the actual number of nodes in the test network, the graph structure is dynamically reconstructed through the adjacency matrix reset interface of the graph encoder, and the corresponding action masks are regenerated. Each agent uses local observations, node context output by the graph encoder, and static attributes as input. The shared Actor network infers the action probability distribution, samples it, outputs phase commands, and executes them. The global Critic network is not enabled during the testing phase. The signal control performance of the model is then tested in three network scenarios: a 2×2 heterogeneous training network, a 4×4 homogeneous network, and a real heterogeneous network with 14 intersections. The corresponding generalized spatiotemporal trajectory diagrams are shown below. Figure 5 , Figure 6 and Figure 7 As shown. In other embodiments, the target road network can be a road network with arbitrary node size and phase configuration.

[0056] Example 3 An electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method described in Embodiment 1 or 2 when executing the computer program. The electronic device in this embodiment may be a signal controller at an intersection, an edge computing server in a regional traffic control center, or an onboard computing platform.

[0057] Example 4 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in Embodiment 1 or 2.

[0058] Example 5 A computer program product includes a computer program that, when executed by a processor, implements the method as described in Example 1 or 2.

[0059] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0060] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0061] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0062] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0063] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections, characterized in that, Includes the following steps: S1. Abstract the road network into a graph, where nodes correspond to heterogeneous intersections and edges correspond to directional roads connecting the intersections. Extract the static attributes of each intersection as the identity identifier of the corresponding node. S2. Based on the above figure, the multi-intersection signal cooperative control problem is modeled as a distributed partially observable Markov decision process, in which each signal-controlled intersection is an independent agent, each agent obtains its own local observations, and achieves cooperative optimization by sharing global rewards; The spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network, and the dynamic hidden state of each intersection at the current moment is output. By using an attribute-guided heterogeneous diffusion attention mechanism, the static attributes of each node are mapped to node identity embeddings. The dynamic hidden states of neighboring nodes are then aggregated with attention weighting based on these node identity embeddings to obtain enhanced node representations. Each agent takes its own local observations, the enhanced node representations, and the static attributes as inputs, and outputs the signal light phase command for the next stage of the intersection through the shared Actor network; The shared Actor network and the global Critic network are trained using a centralized training-distributed execution framework. During the training phase, the global Critic network estimates the global value based on the dynamic hidden states of all nodes, and uses the trajectories collected by each agent to jointly update the parameters of the shared Actor network and the global Critic network. During the execution phase, each agent independently outputs the traffic light phase command through the shared Actor network, and the global Critic network is not enabled.

2. The generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections according to claim 1, characterized in that, In S1, The static attribute is a five-dimensional vector determined by the following formula. : in, This indicates the number of lane turning connections currently controlled by the intersection's traffic lights. and These represent the total number of inbound lanes and the total number of outbound lanes, respectively. Indicates the number of active phases at the current intersection. This indicates the number of lanes contained in the current green light phase p. To prevent extremely small positive numbers with a denominator of zero, , , These are the corresponding normalization constants.

3. The generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections according to claim 1, characterized in that, In S2, The enhanced node representation obtained through the attribute-guided heterogeneous diffusion attention mechanism specifically includes: The static attributes are mapped to node identities using a multilayer perceptron and embedded as keys. The query value is obtained by linearly projecting the dynamic hidden state. ; Calculate the heterogeneous similarity matrix between nodes using scaled dot product attention. ,in, The dimension of the attention head; Introducing a mask matrix When node i is adjacent to node j or i=j ,otherwise The mask matrix is ​​applied to the heterogeneous similarity matrix and normalized to obtain the attention weights. ; The dynamic hidden state of neighboring nodes is determined based on the attention weights. Weighted aggregation is performed to obtain enhanced heterogeneous spatiotemporal features. : , where N(i) is the set of neighboring nodes of node i.

4. The generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections according to claim 1, characterized in that, During the training phase, the parameters of the shared Actor network and the global Critic network are jointly updated using the trajectories collected by each agent through a proximal policy optimization algorithm. The hybrid reward during the intensive training phase is a weighted sum of the agent's local reward and the global average reward, calculated as follows: in, Let α be the local reward of agent i at time t, N be the total number of agents, and α be the weight coefficient.

5. The generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections according to claim 1, characterized in that, Also includes S3: The system receives the current traffic light phase and its remaining time corresponding to the traffic light phase command, obtains the real-time number of vehicles in the queue, the physical length of the queue, the distance to the vehicle in front, and the speed of the vehicle in front in the lane where the target intelligent connected vehicle is currently located, and calculates and smooths the target speed of the target intelligent connected vehicle in real time through space-time coupled target planning and issues the plan for execution.

6. The generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections according to claim 1, characterized in that, When deploying to the target road network after training is completed, the trainable weights of the shared Actor network are frozen. Based on the actual number of nodes in the target road network, the graph structure is dynamically reconstructed through the adjacency matrix reset interface, and the corresponding action mask is regenerated. Each agent takes its local observations, the enhanced node representations, and the static attributes as inputs, and infers the action probability distribution through the shared Actor network. After sampling, it outputs traffic light phase commands and issues them for execution. The global Critic network is not enabled during the deployment phase to achieve zero-sample transfer to the target road network without retraining.

7. A generalizable graph multi-agent reinforcement learning signal control system for heterogeneous intersections, characterized in that, include: The graph construction module is used to abstract the road network into a graph, where nodes correspond to heterogeneous intersections, edges correspond to directional roads connecting the intersections, and the static attributes of each intersection are extracted as the identity identifiers of the corresponding nodes. The signal decision module is used to model the multi-intersection signal cooperative control problem as a distributed partially observable Markov decision process based on the above diagram, where each signal-controlled intersection is an independent agent and each agent acquires its own local observations. The spatiotemporal features of each node are extracted using a diffusing convolutional recurrent network, and the dynamic hidden state of each intersection at the current moment is output. By using an attribute-guided heterogeneous diffusion attention mechanism, the static attributes of each node are mapped to node identity embeddings. The dynamic hidden states of neighboring nodes are then aggregated with attention weighting based on these node identity embeddings to obtain enhanced node representations. Each agent takes its own local observations, the enhanced node representations, and the static attributes as inputs, and outputs the signal light phase command for the next stage of the intersection through the shared Actor network; The shared Actor network and the global Critic network are trained using a centralized training-distributed execution framework; The trajectory smoothing module receives the current traffic light phase and its remaining time, and calculates and smooths the target speed of the intelligent connected vehicle in real time through space-time coupled target planning and sends it out for execution.

8. An electronic device, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the generalizable graph multi-agent reinforcement learning signal control method for heterogeneous intersections as described in any one of claims 1 to 6.

Images