A traffic signal cooperative control method and system based on multi-scale spatial attention
By employing a traffic signal cooperative control method based on multi-scale spatial attention, and utilizing an enhanced VAE encoder and reinforcement learning algorithm, a global traffic state representation is generated. This solves the modeling problem of long-distance and multi-scale features in large-scale urban networks, and achieves efficient traffic signal cooperative control and improved traffic efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGSHA UNIVERSITY
- Filing Date
- 2025-09-28
- Publication Date
- 2026-06-05
Smart Images
Figure CN121545371B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent transportation Internet of Things application technology, specifically relating to a traffic signal collaborative control method and system based on multi-scale spatial attention. Background Technology
[0002] Existing large-scale urban transportation networks have the following characteristics:
[0003] (1) Decentralized control and locality of observation: In large-scale traffic networks, each traffic light (or agent) is usually controlled independently, and its sensors can only acquire local traffic information of its own intersection and the surrounding area. This "partially observable" characteristic makes it impossible for a single agent to perceive the global traffic situation.
[0004] (2) Complex spatial dependence of traffic flow: Urban traffic is a complex dynamic system. Congestion in a region may not be caused by local traffic, but by a chain reaction of traffic events (such as accidents or traffic surges) at another intersection several kilometers away. This "long-distance dependence" characteristic requires the controller to have a global view in order to make optimal decisions.
[0005] (3) Multi-scale characteristics of traffic patterns: Traffic congestion exhibits multi-scale characteristics in space. There are both "micro" patterns of queuing in a single lane and "macro" patterns of slow traffic over multiple blocks or the entire area. Controllers need to be able to understand and process these different scale traffic patterns simultaneously.
[0006] In view of the characteristics of the current traffic signal control, the following existing technologies exist:
[0007] The first category is communication methods based on graph neural networks (GNNs), such as CoLight. This type of method models the road network as a graph, with each intersection as a node, and then uses the graph neural network to "pass messages" between adjacent nodes, allowing agents to share local information to achieve local coordination. However, this type of existing technology still has shortcomings: (1) Inefficient long-distance information transmission: Information needs to be propagated in the graph in multiple rounds and hop-by-hops to reach distant nodes. This may cause information to be distorted or delayed during transmission, making it difficult to effectively capture the direct and instantaneous associations between distant nodes. (2) Scalability and computational bottlenecks: In large-scale, dense urban road networks, the number of nodes and edges in the graph is huge, and the computational overhead of message passing will increase sharply, limiting the feasibility of its real-time application.
[0008] The second category is global context learning methods based on standard convolutional neural networks (CNNs), such as MacLight. This type of method aggregates the local information of all intersections in the entire road network into a two-dimensional "traffic state map", and then uses a CNN-based autoencoder (such as VAE) to learn a compressed representation of the global state. However, this type of existing technology still has shortcomings: (1) Fixed local receptive field: Standard CNNs operate with convolutional kernels of fixed size, and their receptive field is local and fixed. This makes it difficult to directly model the intrinsic relationship between two pixels that are far apart in the image (i.e., two intersections that are far apart in the road network), and it cannot effectively solve the "long-distance dependency" problem. (2) Single-scale feature extraction limitation: A single-size convolutional kernel can only effectively capture traffic patterns at a specific scale. It cannot simultaneously and efficiently handle microscopic local congestion and macroscopic regional traffic conditions, and its ability to capture "multi-scale features" is insufficient. Summary of the Invention
[0009] This invention provides a traffic signal cooperative control method and system based on multi-scale spatial attention, aiming to solve the problem of how to generate and provide a global traffic state representation that can accurately capture long-distance, multi-scale spatial dependencies for each independent traffic light controller in a large-scale urban traffic network, so as to achieve efficient cooperative control and improve the traffic efficiency of the entire road network.
[0010] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0011] A traffic signal cooperative control method based on multi-scale spatial attention includes:
[0012] Acquire the status data of each intersection in the urban road network, and aggregate the status data of all intersections based on geographical location to obtain a global status map;
[0013] An enhanced VAE encoder is used to process the global state graph to generate a global state representation vector; wherein, the enhanced VAE encoder includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit;
[0014] Each intersection employs an agent based on reinforcement learning algorithms to generate local traffic control signals, including the timing of signal light switching, based on the global state representation vector and local state data.
[0015] Furthermore, the intersection status data includes: vehicle queue length, vehicle density, current traffic light phase, and waiting time for each approach lane.
[0016] Furthermore, before aggregating the state data, the acquired state data is preprocessed and aligned in time for different intersecting state data, and then the obtained state data is aggregated to obtain a global state graph; the preprocessing includes: noise filtering, outlier detection and correction, and missing data completion.
[0017] Furthermore, the aggregation of state data to obtain a global state map specifically involves arranging the state data of N intersections deployed in the urban road network according to their geographical location in the physical world into a... 3D tensor Where H and W represent the grid height and width of the road network, and C represents the dimension of the status data for each intersection.
[0018] Furthermore, the spatial self-attention unit is used to compute the input feature map. The interdependence between any two spatial locations is determined by the following operation: based on the feature map. Generate three matrices: query Q, key K, and value V, with two-dimensional spatial location codes added to Q and K; then calculate... We obtain an attention-weighted output feature map. .
[0019] Furthermore, the multi-scale convolutional unit internally sets up multiple convolutional paths in parallel, each path using a convolutional kernel of a different size. Then, the output feature maps of all parallel paths are concatenated along the channel dimension. Finally, a 1x1 convolutional layer is used to fuse information and reduce the dimensionality of the concatenated feature map, resulting in a comprehensive feature map that simultaneously contains multi-scale information. .
[0020] Furthermore, the global state representation vector is reconstructed using a VAE decoder. The enhanced VAE encoder and VAE decoder are trained end-to-end by minimizing the reconstruction loss and KL divergence loss to obtain the enhanced VAE encoder used to generate the global state representation vector.
[0021] Furthermore, the agent based on the reinforcement learning algorithm adopts the PPO agent, which uses generalized advantage estimation (GAE) to guide policy updates and value function optimization. The Actor network takes the local intersection state data as input and outputs the probability of traffic light signal phase selection. The Critic network takes the global state representation vector and the local intersection state data as input and outputs an estimated state value to guide the policy optimization of the Actor network.
[0022] A traffic signal cooperative control system based on multi-scale spatial attention includes a central processing unit and multiple terminal controllers deployed at intersections.
[0023] The central processing unit is used to: aggregate the status data uploaded from each intersection of the urban road network based on geographical location to obtain a global status map; and use an enhanced VAE encoder to process the global status map, generate a global status representation vector, and distribute it to each terminal controller; wherein, the enhanced VAE encoder includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit;
[0024] Each terminal controller has a built-in reinforcement learning agent that generates local traffic control signals, such as the timing of switching traffic lights, based on the global state representation vector and the state data of the local intersection.
[0025] Furthermore, the traffic signal cooperative control system is used to implement the traffic signal cooperative control method described in any of the above embodiments.
[0026] This invention presents a traffic signal cooperative control method and system based on multi-scale spatial attention. In representing the global state of the road network: it utilizes a spatial self-attention mechanism to overcome the limitations of the local receptive field in traditional CNNs, achieving accurate modeling of long-distance dependencies in the traffic network; and employs multi-scale convolution processing to simultaneously capture fine-grained and coarse-grained traffic patterns from individual intersections to the entire region. In decision-making at road network intersections, a VAE-PPO integrated architecture is adopted, organically combining the advanced representation learning technology of VAEs with an efficient multi-agent reinforcement learning framework. This globally information-guided distributed control achieves global-level cooperative control while maintaining the efficiency of terminal decision-making.
[0027] Therefore, the present invention, based on the above-mentioned core technological innovations, has the following technical effects:
[0028] (1) Global perception capability: It can capture network-level traffic conditions and congestion evolution trends.
[0029] (2) Dynamic adaptability: It can respond in real time to dynamic changes in traffic flow and temporary changes in network topology (such as road closures).
[0030] (3) Scalability: The modular design enables it to flexibly support the deployment of transportation networks of different sizes and structures.
[0031] (4) Computational efficiency: The architecture of "centralized training and distributed execution" ensures the performance requirements of real-time control.
[0032] (5) Robustness: Multi-scale feature extraction enhances the system’s ability to resist sensor noise and abnormal traffic events. Attached Figure Description
[0033] Figure 1This is an architecture diagram of the traffic signal cooperative control method and system according to embodiments of this application.
[0034] Figure 2 This is a flowchart illustrating the implementation of the traffic signal coordinated control method in this application.
[0035] Figure 3 This is a timing diagram of the interaction between multiple entities within the traffic signal cooperative control system of this application embodiment.
[0036] Figure 4 This is a modular structure diagram of the traffic signal cooperative control system according to an embodiment of this application. Detailed Implementation
[0037] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0038] Example 1
[0039] This embodiment provides a traffic signal cooperative control method based on multi-scale spatial attention, referencing... Figures 1-3 As shown, it includes:
[0040] Step 1: Obtain the status data of each intersection in the city's road network, and aggregate the status data of all intersections based on geographical location to obtain a global status map.
[0041] First, real-time status data of each intersection is received from N traffic signal controllers (intelligent agents) deployed in the urban road network. This data includes, but is not limited to: vehicle queue length, vehicle density, current traffic light phase, and waiting time for each approach lane. These local traffic information are collected in real time by sensors (such as cameras and geomagnetic coils) deployed at each intersection, serving as the real-time status data for that intersection.
[0042] Before aggregating the state data, the acquired state data is preprocessed and aligned in time for different cross-state data. Then, the obtained state data is aggregated to obtain a global state map. The preprocessing includes noise filtering, outlier detection and removal, and data calibration.
[0043] The aggregation of state data to obtain a global state map specifically involves arranging the state data of N intersections deployed in the urban road network according to their geographical location in the physical world into a map. The three-dimensional tensor (i.e., the global state graph) ); where H and W represent the grid height and width of the road network, and C represents the dimension of the status data for each intersection.
[0044] Step 2: Use the enhanced VAE encoder to process the global state graph and generate a global state representation vector.
[0045] The enhanced VAE encoder of the present invention includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit.
[0046] 2.1) Initial Feature Extraction Unit:
[0047] Preliminary feature extraction is performed on the input global state map Xt using one or more standard convolutional blocks (containing convolutional layers, normalization layers, and activation functions) to obtain an initial feature map. .
[0048] 2.2) Spatial Self-Attention Unit (used to solve long-distance dependency problems):
[0049] This unit receives the feature map. And calculate the interdependencies between any two spatial locations (intersections), specifically by: based on the feature map Generate three matrices: query Q, key K, and value V, with two-dimensional spatial location codes added to Q and K; then calculate... We obtain an attention-weighted output feature map. The output feature map Each position incorporates information from all other positions globally, and the weights are determined by the correlation between them.
[0050] Through this spatial self-attention unit, even two intersections that are far apart at both ends of the road network can be directly connected, thus efficiently capturing long-distance spatial dependencies.
[0051] 2.3) Multi-Scale Convolution Unit (used to solve multi-scale feature problems):
[0052] This unit receives the output feature map from the spatial self-attention unit. First, multiple convolutional paths are set up in parallel internally, each using convolutional kernels of different sizes (3x3, 5x5, 7x7). All convolutional operations use 'same' padding to maintain the feature map size. The 3x3 convolutional kernel is used to capture fine-grained local traffic patterns, such as local congestion, while the 5x5 and 7x7 convolutional kernels are used to capture larger-scale, more macroscopic traffic aggregation patterns, such as regional traffic trends. Then, the output feature maps of all parallel paths are concatenated along the channel dimension. Finally, a 1x1 convolutional layer is used to fuse information and reduce the dimensionality of the concatenated feature maps, resulting in a comprehensive feature map that simultaneously contains multi-scale information. .
[0053] 2.4) Latent variable generation unit:
[0054] This unit will integrate feature maps The data is flattened and passed through a fully connected layer to ultimately generate a latent distribution parameter describing the global traffic state: a mean vector. Sum of logarithmic variance vector Among them, the mean vector This is the highly condensed "global state representation vector" ultimately generated by this method.
[0055] Step 3: At each intersection, an agent based on a reinforcement learning algorithm generates local traffic control signals based on the global state representation vector and local state data: the time for switching traffic lights.
[0056] In a preferred embodiment, the agent based on the reinforcement learning algorithm is a PPO agent, which uses generalized advantage estimation (GAE) to guide policy updates and value function optimization. The Actor network takes the local intersection state data as input and outputs the probability of traffic light signal phase selection. The Critic network takes the global state representation vector and the local intersection state data as input and outputs an estimated state value to guide the policy optimization of the Actor network.
[0057] 3.1) Enhance the Critic network:
[0058] For each agent i, when constructing the input to its Critic network (value network), the state data of the agent's local intersection is used. Global state representation vector shared globally Concatenate the data; the input format is: .
[0059] Because the Critic network obtains global context information, it can make a more accurate assessment of the value of the current state (local + global), thereby providing more precise guidance signals (such as the advantage function) to the Actor network during training.
[0060] 3.2) Actor Network:
[0061] The input to the Actor network (policy network) of each agent i is only its local observation data, that is, the state data of the agent's local intersection. This ensures that when making actual decisions, the Actor has a fast response speed, low computational overhead, and does not rely on the real-time transmission of global information.
[0062] In a preferred embodiment, the enhanced VAE encoder is trained as follows: the global state representation vector is reconstructed using a VAE decoder; and the enhanced VAE encoder and VAE decoder are trained end-to-end by minimizing the reconstruction loss and KL divergence loss to ensure the global state representation vector... It can effectively compress and represent global state information. This results in an enhanced VAE encoder for generating global state representation vectors.
[0063] The training method for PPO agents is as follows: All agents' Actor and Critic networks are trained synchronously. The reward function is typically based on traffic efficiency metrics, such as negative total intersection waiting time or total queue length. Higher reward values indicate better control performance. Under the precise guidance of the Critic, although the Actor only considers local information, its strategy gradually evolves to be more globally coordinated, ultimately improving the overall traffic efficiency of the road network.
[0064] Example 2
[0065] This embodiment provides a traffic signal cooperative control system based on multi-scale spatial attention, used to implement the traffic signal cooperative control method described in Embodiment 1, including a central processing unit and multiple terminal controllers deployed at the intersection.
[0066] The central processing unit includes a state aggregation module and a representation learning module. The state aggregation module aggregates the state data uploaded from each intersection of the urban road network based on geographical location to obtain a global state map. The representation learning module processes the global state map using an enhanced VAE encoder to generate a global state representation vector. The central processing unit then distributes the generated global state representation vector to each terminal controller via a communication network.
[0067] The enhanced VAE encoder includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit.
[0068] Each terminal controller has a built-in reinforcement learning agent that generates local traffic control signals, such as the timing of switching traffic lights, based on the global state representation vector and the state data of the local intersection.
[0069] like Figure 4 As shown in the figure. The traffic signal cooperative control system in this embodiment is also equipped with a data sensing module, an execution output module, and an optimization feedback module.
[0070] The data sensing module is responsible for collecting, preprocessing, and standardizing various sensor data from the physical traffic environment, serving as the system's data input source. It includes the following sub-modules:
[0071] (1) Sensor interface module:
[0072] Function: Communicates with various sensor devices deployed at intersections to obtain raw traffic data.
[0073] Supported devices: cameras, inductive loops, microwave radar, lidar, etc.
[0074] Data types: raw image, vehicle detection bounding box, velocity, density, etc.
[0075] (2) Data preprocessing module:
[0076] Function: Cleans and performs preliminary processing on raw sensor data to improve data quality.
[0077] Processing includes: noise filtering, outlier detection and removal, and data calibration.
[0078] Output format: Standardized numerical data.
[0079] (3) State standardization module:
[0080] Function: Unifies data from different sources and in different formats into a standard internal format.
[0081] Standardization includes: data range normalization (e.g., [0, 1]), timestamp synchronization, and uniformity of physical units.
[0082] Quality assurance: Perform data integrity checks and cross-source data consistency verification.
[0083] (4) Local observation generator:
[0084] Function: Integrates processed data into a standardized observation vector for a single intersection. .
[0085] Observation content includes: signal phase, lane density, queue length, green light time status, etc.
[0086] Output specification: A feature vector of fixed dimensions.
[0087] The output module is responsible for converting the traffic control signals generated by the PPO agent into specific signal control commands in the physical world and monitoring them.
[0088] The optimization feedback module calculates rewards and gradients based on execution feedback and updates the parameters of all learnable modules, forming a learning loop.
[0089] The network structure and working principle of each unit module and reinforcement learning agent of the enhanced VAE encoder in this embodiment are the same as those described in Embodiment 1, and will not be repeated in this embodiment.
[0090] The key data flow and control flow of the above two embodiments of the present invention can be summarized as follows:
[0091] 1. Main data flow path:
[0092] (1) Sensing data flow: physical layer → data acquisition layer → global state diagram.
[0093] (2) Representation of learning flow: Global state graph → VAE encoder → Global representation.
[0094] (3) Decision data flow: global representation + local observation → PPO agent → control command.
[0095] (4) Feedback optimization flow: Reconstruction loss → VAE parameter update.
[0096] 2. Control and coordination mechanism:
[0097] (1) Spatial coordination: Information exchange and coordination between remote intersections are achieved through spatial self-attention mechanism.
[0098] (2) Time coordination: Maintain the synchronization of the entire system in decision-making through the time-series update of the global state.
[0099] (3) Decision coordination: Through the shared global representation, the decisions of each agent can take into account the global situation, thereby maintaining consistency and coordination.
[0100] 3. Optimize the feedback loop:
[0101] (1) VAE optimization loop: The encoder and decoder are optimized in a closed loop using ELBO loss.
[0102] (2) PPO optimization loop: The Actor network and the Critic network perform closed-loop optimization through policy gradient.
[0103] (3) End-to-end optimization: The VAE module and the PPO agent are jointly trained to achieve the optimal overall system performance.
[0104] The present invention addresses the limitations of traditional methods by employing corresponding technical means, summarized as follows:
[0105] (1) Limitations of traditional methods: GNN method requires multi-hop propagation, which results in low information transmission efficiency; CNN method's fixed receptive field cannot directly capture long-distance associations.
[0106] This invention introduces a spatial self-attention mechanism, which allows the model to directly model the spatial relationship between any two intersections in a single-step calculation, efficiently capturing global dependencies.
[0107] (2) Limitations of traditional methods: A single-size convolution kernel cannot simultaneously handle traffic phenomena of different granularities (such as single-lane queuing vs. regional congestion).
[0108] This invention employs a multi-scale convolutional architecture to process features at different spatial scales in parallel and then fuse them to achieve a comprehensive understanding of complex traffic patterns.
[0109] (3) Limitations of traditional methods: Completely centralized control has high computational complexity and poor real-time performance; Completely distributed control lacks a global perspective and is prone to local optima rather than global optima.
[0110] This invention adopts a paradigm of "centralized learning and distributed execution." Implicit coordination among agents is achieved through a shared global representation vector, while maintaining the high efficiency of each agent making decisions based on local information.
[0111] (4) Limitations of traditional methods: Strategies based on fixed rules or offline optimization are difficult to adapt to real-time fluctuations in traffic flow and sudden events.
[0112] This invention employs an end-to-end reinforcement learning framework, enabling the system to continuously optimize and adjust its strategies through ongoing interaction with the environment, thereby automatically adapting to dynamically changing environments.
[0113] The technical solution of this invention is applicable to the following intelligent traffic management scenarios:
[0114] 1. Intelligent control of urban traffic networks: (1) Coordinated control of large-scale urban road intersections. (2) Signal optimization of main roads in commercial and residential areas. (3) Traffic management of closed areas such as industrial parks and campuses.
[0115] 2. Dynamic traffic flow management: (1) Traffic flow control during morning and evening peak hours. (2) Emergency traffic diversion for sudden events (such as traffic accidents). (3) Optimization of alternative routes when roads are under construction or closed.
[0116] 3. Smart city infrastructure: (1) It can serve as a core module of a smart city traffic management system. (2) It can be integrated with the city's big data platform. (3) It supports the collaborative passage of future autonomous vehicles.
[0117] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A traffic signal cooperative control method based on multi-scale spatial attention, characterized in that, include: Acquire the status data of each intersection in the urban road network, and aggregate the status data of all intersections based on geographical location to obtain a global status map; The status data of the intersection includes: the queue length, vehicle density, current traffic light phase, and waiting time for each approach lane; Specifically, the aggregation of state data to obtain the global state map involves arranging the state data of N intersections deployed in the urban road network according to their geographical location in the physical world into a map. 3D tensor Where H and W represent the grid height and width of the road network, and C represents the dimension of the status data for each intersection; An enhanced VAE encoder is used to process the global state graph to generate a global state representation vector; wherein, the enhanced VAE encoder includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit; Specifically, the global state representation vector is reconstructed through a VAE decoder, and the enhanced VAE encoder and VAE decoder are trained end-to-end by minimizing the reconstruction loss and KL divergence loss to obtain the enhanced VAE encoder used to generate the global state representation vector. Each intersection employs an agent based on reinforcement learning algorithms to generate local traffic control signals, including the timing of signal light switching, based on the global state representation vector and local state data. The agent based on the reinforcement learning algorithm is a PPO agent, which uses generalized advantage estimation (GAE) to guide policy updates and value function optimization. The Actor network takes the local intersection state data as input and outputs the probability of traffic light signal phase selection. The Critic network takes the global state representation vector and the local intersection state data as input and outputs an estimated state value to guide the policy optimization of the Actor network.
2. The traffic signal cooperative control method according to claim 1, characterized in that, Before aggregating the state data, the acquired state data is preprocessed and aligned in time for different cross-state data. Then, the obtained state data is aggregated to obtain a global state graph. The preprocessing includes noise filtering, outlier detection and correction, and missing data completion.
3. The traffic signal cooperative control method according to claim 1, characterized in that, The spatial self-attention unit is used to compute the input feature map. The interdependence between any two spatial locations is determined by the following operation: based on the feature map. Generate three matrices: query Q, key K, and value V, with two-dimensional spatial location codes added to Q and K; then calculate... We obtain an attention-weighted output feature map. .
4. The traffic signal cooperative control method according to claim 1, characterized in that, The multi-scale convolutional unit internally sets up multiple convolutional paths in parallel, each path using a convolutional kernel of a different size. Then, the output feature maps of all parallel paths are concatenated along the channel dimension. Finally, a 1x1 convolutional layer is used to fuse information and reduce the dimensionality of the concatenated feature map, resulting in a comprehensive feature map that simultaneously contains multi-scale information. .
5. A traffic signal cooperative control system based on multi-scale spatial attention, characterized in that, It includes a central processing unit and multiple terminal controllers deployed at the intersection; The central processing unit is used to: aggregate the status data uploaded from each intersection of the urban road network based on geographical location to obtain a global status map; and use an enhanced VAE encoder to process the global status map, generate a global status representation vector, and distribute it to each terminal controller; wherein, the enhanced VAE encoder includes: an initial feature extraction unit, a spatial self-attention unit, a multi-scale convolution unit, and a latent variable generation unit; The status data of the intersection includes: the queue length, vehicle density, current traffic light phase, and waiting time for each approach lane; Specifically, the aggregation of state data to obtain the global state map involves arranging the state data of N intersections deployed in the urban road network according to their geographical location in the physical world into a map. 3D tensor Where H and W represent the grid height and width of the road network, and C represents the dimension of the status data for each intersection; Specifically, the global state representation vector is reconstructed through a VAE decoder, and the enhanced VAE encoder and VAE decoder are trained end-to-end by minimizing the reconstruction loss and KL divergence loss to obtain the enhanced VAE encoder used to generate the global state representation vector. Each terminal controller has a built-in reinforcement learning agent used to generate local traffic control signals based on the global state representation vector and the state data of the local intersection: the timing of switching traffic lights; The reinforcement learning agent is a PPO agent, which uses generalized advantage estimation (GAE) to guide policy updates and value function optimization. The Actor network takes the local intersection state data as input and outputs the probability of traffic light signal phase selection. The Critic network takes the global state representation vector and the local intersection state data as input and outputs an estimated state value to guide the Actor network's policy optimization.