Adaptive traffic signal control method based on reinforcement learning and self-attention mechanism

Abstract

The application relates to the technical field of traffic signal control, and particularly discloses an adaptive traffic signal control method based on reinforcement learning and a self-attention mechanism, which comprises the following steps: establishing a Markov decision process model and a policy neural network based on a deep reinforcement learning algorithm; building a deep Q network and a self-attention mechanism based on the Markov decision process model; evaluating the policy and updating the neural network of the policy neural network, an encoder and a decoder; performing multiple iterative training on the neural network; obtaining a trained neural network model; and generating real-time traffic signal control signals according to the trained neural network model. The method can gradually learn an optimal policy in a complex traffic flow. In addition, the algorithm adopts an encoder-decoder structure, stores historical trajectories in a replay buffer, and uses proximal policy optimization to update parameters. The method solves the problem that the existing adaptive traffic signal control method needs a large amount of data for training and requires a large amount of computing resources.

Description

Technical Field

[0001] This invention relates to the field of traffic signal control technology, specifically an adaptive traffic signal control method based on reinforcement learning and self-attention mechanisms. Background Technology

[0002] In recent years, population growth and rapid urbanization have led to a surge in urban travel demand. Traffic congestion is a product of the imbalance between traffic supply (road capacity) and travel demand. Intersections, as key nodes in the urban road network, gradually become bottlenecks in the network's capacity, preventing the continuous improvement of traffic efficiency.

[0003] In the past, to meet the ever-increasing traffic demands of cities, the supply capacity of urban roads was increased by constructing new road infrastructure, addressing the growth in traffic demand from the supply side. However, due to limitations in funding and road space, this is not an economically effective long-term solution. Therefore, under the condition of limited road supply, the focus of research should shift to the demand side of urban traffic, optimizing existing travel demand through traffic management and control measures. Traffic signal control, as the core of urban traffic management and control, can maximize the throughput of intersections, improve the operational efficiency of the urban road network and the capacity of intersections, reduce the frequency and intensity of various traffic conflicts, and thus alleviate urban traffic congestion.

[0004] Currently, many intersections use fixed-time traffic signal systems, which set signal durations based on historical data rather than adjusting them according to real-time traffic demand. Traditional traffic signal control techniques typically use predefined rules derived from expert experience. These methods lack the ability to dynamically adjust signal phases based on instantaneous traffic flow data. Adaptive traffic signal control is particularly important in this context. Adaptive Traffic Signal Control (ATSC) is a hot topic in intelligent transportation systems research. It can optimize traffic flow in regional road networks by dynamically changing signal phases to respond to current traffic conditions, effectively reducing congestion. While ATSC can alleviate urban traffic congestion to some extent, it primarily relies on predictions from fixed traffic models, selection of pre-defined signal control schemes, or real-time predictions of traffic demand based on traffic simulation models, such as the Split, Cycle and Offset Optimization Technique (SCOOT) and the Sydney Coordinated Adaptive Traffic System (SCATS). They can adjust traffic lights based on real-time traffic volume, but they are essentially based on traffic model-driven systems. They require prior calibration of traffic simulation model parameters and design of predefined signal control schemes based on actual traffic scenarios. Due to the dynamic, random, and uncertain nature of the actual traffic environment, the applicability of simulation model parameters and predefined signal control schemes is poor.

[0005] While traditional reinforcement learning-based ATSC algorithms are theoretically considered capable of adjusting traffic signals, they face several limitations in practical applications. First, these algorithms often struggle to accurately capture the correlations between time series when dealing with dynamic changes in traffic flow and the high-dimensionality of information, and they also fail to fully understand complex traffic scenarios. This lack of in-depth insight leads to significant non-stationarity and instability during training. Second, most ATSC algorithms require a large amount of data to achieve efficient policy learning, which not only increases the dependence on computational resources but also significantly prolongs the learning process. Third, these algorithms are typically optimized in specific training environments, making them difficult to generalize and transfer to new or unknown environments, thus limiting their application in broader and more complex traffic environments. Due to these factors, RL algorithms are feasible for simple scenarios, but their effectiveness and adaptability in complex real-world traffic conditions are greatly limited. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive traffic signal control method based on reinforcement learning and self-attention mechanisms to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an adaptive traffic signal control method based on reinforcement learning and self-attention mechanisms, the method comprising:

[0008] A Markov decision process model and a policy neural network are established based on a deep reinforcement learning algorithm. The observation space constraining the Markov decision process is determined based on the sum of various traffic flow information at the intersection. The phase of the traffic lights at the intersection constrains the behavior space of the Markov decision process. The reward function of the Markov decision process is constrained based on the weighted sum of the total number of vehicles, speed, queue length, and waiting time.

[0009] A deep Q-network and self-attention mechanism were built based on the Markov decision process model;

[0010] The policy is evaluated and the neural networks of the policy neural network, encoder, and decoder are updated;

[0011] The neural network is trained iteratively multiple times to obtain the trained neural network model;

[0012] Based on the trained neural network model, real-time traffic signal control signals are generated.

[0013] As a further aspect of the present invention, the step of evaluating the policy and updating the neural networks of the policy neural network, encoder, and decoder specifically includes:

[0014] Sample the agent's trajectory;

[0015] The current strategy is evaluated based on the reward function and the value function;

[0016] The neural network parameters of the policy network, encoder, and decoder are optimized and updated based on the proximal policy.

[0017] As a further aspect of the present invention, the step of evaluating the current strategy based on the reward function and the value function specifically includes:

[0018] Calculate the reward function;

[0019] Calculate the state value function and the behavior value function based on the reward function and the observed probability distribution;

[0020] Calculate the advantage function based on the state value function and the behavior value function;

[0021] The current strategy is evaluated based on the advantage function results.

[0022] As a further aspect of the present invention, the encoding process of the encoder specifically includes:

[0023] The observations are processed through an embedding layer;

[0024] The processed data is then connected via a self-attention mechanism and residual connections.

[0025] The observation information is further processed by a multilayer perceptron to obtain a representation.

[0026] As a further aspect of the present invention, the decoding process of the decoder specifically includes:

[0027] The initial action flag is processed through an embedding layer;

[0028] The processed data is then processed through the first self-attention mechanism;

[0029] The data and observation information obtained from the first self-attention mechanism are processed by the second self-attention mechanism and the multilayer perceptron to obtain the action representation.

[0030] As a further aspect of the present invention, the observation space is o= The observed subset is ,in Indicates time Lane The total number of vehicles in the country Indicates time Lane Medium speed below The average waiting time for vehicles Indicates that in the lane The waiting time for vehicle m; Indicates time Lane queue length, Indicates time Lane The average speed of vehicles in the middle, Indicates that in the lane The speed of vehicle m in the middle. This indicates the maximum permitted speed for that lane;

[0031] Action space is The intelligent agent at time step After observation Execute actions That is, a traffic signal phase ,implement The intelligent agent decides Lanes that are permitted or prohibited at all times and phase maintenance The time.

[0032] As a further aspect of the present invention, the reward function consists of the following parts:

[0033] , , , ;

[0034] ;

[0035] in This indicates the average waiting time for vehicles. This indicates the average queue length of vehicles. This indicates the average speed of the vehicle. Indicates the total number of vehicles. Indicates the total number of lanes. These are the weighting coefficients for the four reward sub-items.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] Intersection-Transformer leverages the Transformer algorithm to process data in parallel, thus helping to reduce training time. Furthermore, its computationally efficient nature allows it to handle larger state and action spaces. Intersection-Transformer's self-attention mechanism captures dynamic changes in the environment, addressing non-stationarity and training instability. It also captures details of general feature representations, potentially improving the model's generalization ability in new environments. We also introduce PPO, which uses truncated probability ratios to limit policy update steps, preventing large policy changes during training and increasing the stability of the training process.

[0038] By using Markov Decision Process (MDP) to refine the modeling of the ATSC problem, and by carefully designing relevant parameters such as the observation space, action space, and reward function, we can effectively solve the problems of excessive training time and low training efficiency of other ATSC methods.

[0039] Extensive experiments were conducted in various traffic scenarios. Experimental results demonstrate that our IT consistently performs exceptionally well in alleviating traffic congestion and improving traffic system efficiency. The advantages of the Intersection-Transformer algorithm become increasingly apparent as intersection complexity increases. Under the most complex traffic conditions, the publicly available method outperforms the traditional reinforcement learning-based ATSC algorithm by more than 40% in handling traffic flow, such as reducing vehicle waiting times and intersection queue lengths. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0041] Figure 1 This is a flowchart illustrating an adaptive traffic signal control method based on reinforcement learning and self-attention mechanisms, provided in an embodiment of the present invention.

[0042] Figure 2 This is an architecture diagram of an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism provided in an embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the steps of evaluating a policy and updating the neural networks of the policy neural network, encoder, and decoder, as provided in an embodiment of the present invention.

[0044] Figure 4 This is an architecture diagram illustrating the steps for evaluating a policy and updating the neural networks of the policy neural network, encoder, and decoder, as provided in an embodiment of the present invention.

[0045] Figure 5 This is a flowchart illustrating how to evaluate the current strategy step based on a reward function and a value function, as provided in an embodiment of the present invention.

[0046] Figure 6 The reward performance curve provided for embodiments of the present invention.

[0047] Figure 7 A graph showing the reward sub-items (total number of vehicles) provided for embodiments of the present invention.

[0048] Figure 8 The curve of reward sub-items (average waiting time) provided for embodiments of the present invention. Detailed Implementation

[0049] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0050] like Figure 1 , Figure 2 As shown in the embodiment of the present invention, the adaptive traffic signal control method based on reinforcement learning and self-attention mechanism specifically includes:

[0051] Step S100: Establish a Markov decision process model and a policy neural network based on a deep reinforcement learning algorithm, determine the observation space that constrains the Markov decision process based on the sum of various traffic flow information at the intersection, constrain the behavior space of the Markov decision process by the phase of the traffic lights at the intersection, and constrain the reward function of the Markov decision process based on the weighted sum of the total number of vehicles, speed, queue length, and waiting time.

[0052] Step S200: Build a deep Q-network and a self-attention mechanism based on the Markov decision process model;

[0053] Step S300: Evaluate the policy and update the neural networks of the policy neural network, encoder, and decoder;

[0054] Step S400: Perform multiple iterations of training on the neural network to obtain the trained neural network model;

[0055] Step S500: Generate real-time traffic signal control signals based on the trained neural network model.

[0056] This invention discloses an adaptive traffic signal control reinforcement learning algorithm, named Intersection-Transformer, applied to vehicle cooperative control at intersections. This algorithm integrates with Transformer, considering factors such as fairness and efficiency based on traffic information including traffic density and vehicle waiting time. This enables intersection traffic lights to scientifically and rationally adjust their phases and durations adaptively, thereby achieving high traffic efficiency and reducing traffic congestion. The Intersection-Transformer algorithm introduces a Transformer model on top of reinforcement learning, ensuring that the traffic signal control agent can capture the potential connections between different traffic information. The algorithm collects traffic condition-related information from the environment and interacts with the environment based on actions generated by the policy neural network. Simultaneously, it evaluates the policy through a reward function and a value neural network, and uses Proximal Policy Optimization (PPO) to update the parameters of the policy and value neural networks. This solves the problem of existing adaptive traffic signal control methods requiring large amounts of data and computational resources for training. This invention aims to improve the operational efficiency and capacity of urban intersections through a scientifically sound signal control scheme, maximizing intersection throughput and effectively alleviating urban traffic congestion. This invention applies the powerful understanding and reasoning capabilities of LLM to ATSC to adapt to the high dynamics of traffic conditions and the high dimensionality of traffic information. This enables the Intersection-Transformer to progressively learn the optimal strategy in complex traffic flows. Furthermore, this algorithm employs an encoder-decoder structure, storing historical trajectories in an experience replay buffer and using Proximal Policy Optimization (PPO) to update parameters.

[0057] The agent interacts with the environment at the intersection and observes the current environment in real time. The Intersection-Transformer algorithm collects the current policy. The observation space provided by the following environment is Policy neural networks map the state space to the behavior space. The algorithm selects actions from this action space and assigns them to the agent for execution. The agent interacts with the environment based on the acquired actions, obtaining new observation spaces and reward functions, thus forming a closed loop of interaction between the algorithm and the environment. Training ends when the number of policy updates reaches a certain threshold.

[0058] In a preferred embodiment of the present invention, the observation space is used to describe the observation information when the agent interacts with the environment. A reasonable observation space is crucial for the effective training of deep reinforcement learning algorithms. In the ATSC system, it is very important to be able to accurately extract and reconstruct the observation information in the complex dynamic environment of the intersection, because this determines the ability to accurately output appropriate actions.

[0059] The observation space is defined as the sum of observations of lanes in the current scene, i.e., o= Considering that traffic conditions at intersections vary based on dynamic and static traffic information for each lane, the observation subset was set as follows: ,in Indicates time Lane The total number of vehicles in the area; Indicates time Lane Medium speed below The average waiting time for vehicles Indicates that in the lane The waiting time for vehicle m; Indicates time Lane The queue length can be represented by the number of vehicles in the queue; Indicates time Lane The average speed of vehicles in the middle, Indicates that in the lane The speed of vehicle m in the middle. This indicates the maximum permissible speed for that lane. The aforementioned observation space can adapt to complex and randomly changing traffic intersection scenarios, effectively characterizing the traffic conditions at intersections.

[0060] The discrete set of traffic light stages is considered as the action space of the intelligent agent. The intelligent agent at time step After observation Execute actions That is, a traffic signal phase ,implement The intelligence will determine Lanes that are permitted or prohibited at all times and phase maintenance The time.

[0061] In a preferred embodiment of the present invention, the reward function serves as quantitative feedback received by the agent from the environment after performing an action. In this study, the reward function guides the agent to continuously explore and learn the optimal strategy while maximizing the reward. The design of these functions has a crucial impact on the asymptotic performance and convergence speed of deep reinforcement learning algorithms. If the reward function over-optimizes a specific scenario, it may lead to overfitting, reducing the ability of the trained model to adapt to new scenarios. A general rule is that performance metrics should be designed based on the actual goals desired to be achieved in the environment, rather than on how people believe the agent should behave. To comprehensively describe the environment, the reward function focuses on comprehensively improving the traffic efficiency of the intersection, ensuring that the reward system truly reflects the expected results in the environment, rather than merely reflecting how the agent should behave. This design helps avoid overtraining in specific environments while enhancing the model's adaptability and generality in new scenarios. Therefore, considering all factors, the reward function consists of the following parts ( (Indicates the total number of lanes):

[0062] Total number of vehicles ( ): By focusing on the total number of vehicles, the reward function can directly influence and evaluate traffic flow at intersections. Reducing the total number of vehicles at an intersection helps alleviate traffic congestion and make traffic flow smoother. This indicator reflects the intersection's capacity and traffic management efficiency.

[0063] The average speed of the vehicle ( ): Average speed is one of the key indicators for evaluating traffic efficiency. Increasing the average speed of vehicles can reduce traffic congestion, shorten travel time, and thus improve the overall efficiency of an intersection. A higher average speed generally indicates good traffic conditions, with vehicles not needing to stop or slow down frequently.

[0064] Average queue length of vehicles ( ): This indicator reflects the vehicle queuing situation at intersections. Shorter queue lengths mean vehicles can pass through intersections faster, reducing waiting and dwell times, which helps improve traffic flow continuity and reduce congestion. Controlling queue length can effectively optimize traffic flow distribution and traffic light scheduling at intersections.

[0065] Average waiting time for vehicles ( ): Average waiting time is a key indicator of intersection efficiency. Reducing the average waiting time of vehicles at intersections can significantly improve the smoothness and efficiency of traffic flow. Lower waiting times not only reduce inconvenience and anxiety for drivers but also help reduce vehicle fuel consumption and emissions.

[0066] In summary, the reward function is finally designed as follows: ;

[0067] in These are the weighting coefficients for the four reward sub-items.

[0068] The ultimate goal of the proposed algorithm is a weighted sum of these four objectives. This approach can be viewed as a normalization method. This weighted summation method is chosen because the magnitudes of these four factors differ in the initial stages of each environment. Simply adding them together as the reward might cause the agent to focus only on the larger numerical values, leading to the algorithm failing to converge.

[0069] By weighting these four indicators, it is ensured that each indicator has an appropriate proportion in the reward function, thus overcoming the problems of inconsistent dimensions and uneven influence among the indicators. This design helps the algorithm balance the importance of different indicators, avoiding over-optimization of a single indicator while neglecting other equally important ones. Furthermore, this weighting method also helps improve the stability and convergence of the algorithm, enabling it to effectively learn and adapt to various traffic environments, ultimately achieving the goal of reducing congestion and improving traffic efficiency.

[0070] The policy neural network receives observations from the environment. It serves as the input to the neural network and outputs the actions of the agent in the intersection environment. and its probability The input observation values ​​are calculated during the process of outputting the agent's actions. The intelligent agent in time Take action at the time The new state observed later is The intelligent agent will also receive environmental rewards. If the length of the trajectory is The Intersection-Transformer algorithm utilizes the trajectory as .

[0071] like Figure 3 , Figure 4 , Figure 5 As shown, in a preferred embodiment of the present invention, the step of evaluating the policy and updating the neural networks of the policy neural network, encoder, and decoder specifically includes:

[0072] Step S301: Sample the trajectory of the intelligent agent;

[0073] Step S302: Evaluate the current strategy based on the reward function and the value function;

[0074] Step S303: Optimize and update the neural network parameters of the policy network, encoder, and decoder based on the near-end policy.

[0075] The observation space describes the observation information when an agent interacts with its environment. A reasonable observation space is crucial for the effective training of deep reinforcement learning algorithms. In the ATSC system, accurately extracting and reconstructing observation information in the complex dynamic environment of an intersection is essential, as it determines the ability to accurately output appropriate actions. This sub-step collects the agent's observation behavior sequences, and the policy neural network receives observations from the environment. It serves as the input to the neural network and outputs the actions of the agent in the intersection environment. and its probability The input observation values ​​are calculated during the process of outputting the agent's actions. The intelligent agent in time Take action at the time The new state observed later is The intelligent agent will also receive environmental rewards. If the length of the trajectory is The Intersection-Transformer algorithm utilizes the trajectory as .

[0076] Based on the collected agent observation behavior trajectories, the policy is evaluated and optimized. The performance of the current policy is evaluated using a reward function and a value function, respectively. The reward function is used to evaluate the degree to which the policy achieves the agent's cumulative expected reward. The neural network parameters of the policy network, encoder, and decoder are updated through proximal policy optimization.

[0077] In a preferred embodiment of the present invention, the step of evaluating the current strategy based on the reward function and the value function specifically includes:

[0078] Step 3021, calculate the reward function;

[0079] Step 3022: Calculate the state value function and behavior value function based on the reward function and the probability distribution of the observations;

[0080] Step 3023: Calculate the advantage function based on the state value function and the behavior value function;

[0081] Step 3024: Evaluate the current strategy based on the advantage function results.

[0082] In a preferred embodiment of the present invention, the encoding process of the encoder specifically includes:

[0083] The observations are processed through an embedding layer;

[0084] The processed data is then connected via a self-attention mechanism and residual connections.

[0085] The observation information is further processed by a multilayer perceptron to obtain a representation.

[0086] In a preferred embodiment of the present invention, the decoding process of the decoder specifically includes:

[0087] The initial action flag is processed through an embedding layer;

[0088] The processed data is then processed through the first self-attention mechanism;

[0089] The data and observation information obtained from the first self-attention mechanism are processed by the second self-attention mechanism and the multilayer perceptron to obtain the action representation.

[0090] The conceptual structure of the Intersection-Transformer mainly consists of two parts: the encoder and the decoder. The encoder is responsible for receiving observations from the environment. and learn the representation of observations. The decoder is responsible for outputting the actions that the agent will perform. Therefore, after the agent executes a predetermined number of steps, a portion of the historical trajectory is revealed. It will be provided to the PPO to update the strategy. and encoder-decoder networks.

[0091] In strategy evaluation and optimization, the advantage function is used, denoted as... It is used to evaluate the relative advantage of taking a specific action in a given state compared to the average situation. It is the behavioral value function, representing the expected reward of taking an action and following a certain strategy in a given state; while It is the state-value function, representing the expected reward based solely on the state. The calculation formulas are as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] in, Indicates observation The probability distribution, Discount reward function .

[0096] The core idea of ​​the advantage function is to provide a criterion for reinforcement learning algorithms, measuring the difference between taking a specific action and the average performance of the policy. This concept is particularly important in methods such as policy gradient, as it helps us to more precisely adjust policy parameters, thereby optimizing the learning process. Through the advantage function, learning algorithms can identify and prioritize actions that offer higher additional rewards compared to average performance.

[0097] The primary function of an encoder is to extract useful features from the raw environmental input and transform these features into a more abstract and decision-making-friendly internal representation. The encoder's parameters use... It indicates that its function is as follows: Observation values Firstly, through an embedding layer Its function is to map discrete inputs to continuous digital vectors in a low-dimensional space to improve computational efficiency. Then, it uses a self-attention mechanism and residual connections... This is to prevent gradient vanishing and network degradation. Finally, the data is processed by a Multi-Layer Perceptron (MLP) to obtain... This is one of the inputs to the decoder. Furthermore, Also used for calculating values This facilitates subsequent updates within the network. The encoder acts as a judge, so our goal is to make it approximate the value function, which aims to minimize the empirical Bellman error by:

[0098] ;

[0099] in, Represents the loss function. This represents the parameters of the target network, which are non-differentiable and are updated every few time steps. This indicates the number of times the encoder parameters have been updated during training. Represents the reward function, This represents the discount factor. , Let represent the value function of the target network and the current value function, respectively.

[0100] The decoder's task is to translate the representations learned by the encoder into specific action instructions. This is typically achieved through a series of neural network layers that generate the final action output based on the learned features and possible policy guidelines. The decoder is designed to ensure that the output action is adaptable to the needs of the specific task while also reflecting the decision preferences of the policy network under specific environmental conditions.

[0101] Embedded actions (An arbitrary symbol indicating the start of decoding) is passed to the decoder, whose parameters are determined by... The decoder consists of many encoder-like decoder blocks. Each block contains two attention mechanisms and a multilayer perceptron. The first mechanism is responsible for receiving... and generate ,in It is obtained after dimensionality transformation through the embedding layer. In the second attention mechanism, the attention score is calculated as follows: After passing through two MLPs, the action representation is... The output is the probability distribution of the agent's actions, i.e., the policy. Next, we attempt to minimize the clipping PPO objective for training the decoder:

[0102] ;

[0103] ;

[0104] in, This represents the target loss function. It is a ratio that represents the relative probability between the updated new strategy and the old strategy. It is an estimate of the advantage function.

[0105] In the PPO update process, the use of historical trajectories τ is crucial for evaluating and adjusting the current policy. By replaying this historical data, PPO can assess the effectiveness of existing policies and optimize the network parameters of the encoder and decoder accordingly, thereby achieving more accurate and effective control in continuous decision-making processes. This trajectory-based learning approach helps capture temporal dependencies in complex environments, thus improving overall policy performance.

[0106] This patent validates the disclosed method through simulation using the SUMO platform and a reinforcement learning model built on the PyTorch framework. The intersection used in the simulation experiment has 32 lanes, with each direction containing 3 straight lanes, 1 right-turn lane, and 1 left-turn lane. This layout effectively handles traffic flow from all directions. All lanes are 750 meters long, a design that can accommodate a large number of vehicles, especially during peak hours, helping to alleviate traffic congestion. During training, the traffic flow density at the intersection was approximately 4000 vehicles per hour, representing typical urban peak-hour traffic volume, providing a challenging environment for optimizing and testing the algorithm's performance. This high-flow environment simulates real-world urban traffic conditions, making the trained model more suitable for complex real-world traffic management tasks. In the SUMO environment, the shortest simulation time interval (i.e., time step) is... The interval is 1 second. Randomly generated traffic flows will enter the incoming lane and then pass through the intersection within 3600 seconds. The Intersection-Transformer algorithm has been trained 3 million times in this training scenario, with each episode lasting 500 seconds, meaning there are a total of 6000 episodes. In this virtual simulation scenario, the weights of each reward sub-item in the discount reward function are as follows: Every decision The interval is 6 seconds, meaning that every 6 seconds the agent decides whether to maintain the current phase or switch to another phase. If Setting the time too short, such as 1 second, might cause traffic lights to switch too frequently, which is impractical and increases the computational burden. Conversely, if... Setting the value too long will reduce the learning rate of the agent. Detailed parameters and experimental settings for the SUMO simulator, Intersection-Transformer algorithm, and Deep Q-Network (DQN) algorithm are shown in Table 1.

[0107]

[0108] In our performance comparison experiments, we delve into the performance comparison between strategies trained using the Intersection-Transformer algorithm and the DQN algorithm in real-world traffic management. To comprehensively evaluate the effectiveness of the Intersection-Transformer algorithm and demonstrate its applicability under different traffic conditions, we carefully designed two typical test scenarios:

[0109] Simple Scenario: This scenario is designed for low-complexity traffic flow, primarily simulating one-way flow from north to south and from west to east. Each direction includes two lanes, with an overall traffic density set at 600 vehicles per hour. This setting aims to test the algorithm's performance under low-stress conditions, where green light phases switch simply. This scenario reflects low-volume traffic conditions in cities, such as residential areas or roads during off-peak hours.

[0110] Challenge Scenario: Similar to the training environment, this scenario simulates higher traffic density and more complex traffic dynamics. There are four lanes in each direction, with a traffic density of up to 4000 vehicles per hour. Furthermore, the green light phase settings are more complex, incorporating multiple modes. This scenario aims to simulate peak-hour conditions in a city or on a major traffic artery, challenging the algorithm's performance under high pressure and complexity.

[0111] In the above scenario, several experiments were also set up, in which the green light phase was fixed, to compare with the Intersection-Transformer algorithm:

[0112] FIXED-6 / -12 / -18 / -24: These numbers represent the fixed duration of each green light phase. In the FIXED experiment, the green light phases are cycled through at fixed durations to influence traffic conditions, similar to the traffic lights we see at most intersections in real life.

[0113]

[0114] Table 2 presents the final results comparing the performance of fixed green light phase duration, the Intersection-Transformer algorithm, and the DQN algorithm under different traffic scenarios. In simple scenarios, due to relatively low traffic flow, the performance difference between fixed green light phase and adaptive green light phase is not significant. This indicates that even simple fixed-phase control can effectively manage traffic flow in relatively simple and predictable traffic environments.

[0115] However, as intersection size and traffic flow increase, the superiority of the Intersection-Transformer algorithm becomes increasingly apparent. In complex traffic scenarios, the Intersection-Transformer algorithm demonstrates significant performance improvements compared to DQN and fixed-phase control. Specifically, in terms of vehicle number and vehicle speed, the Intersection-Transformer algorithm has achieved a significant performance improvement of approximately 50%. This improvement not only alleviates traffic congestion but also enhances traffic flow efficiency, significantly optimizing vehicle capacity and road utilization.

[0116] The Intersection-Transformer algorithm also performed well in terms of vehicle queue length, reducing it by approximately 30%. This improvement is significant for alleviating traffic congestion and reducing vehicle waiting times, especially during peak traffic hours.

[0117] like Figure 6 As shown in the figure, the curve represents the discounted reward that changes with increasing time steps. From the initial state to the first few thousand steps, the agent explores the impact of various behaviors in different scenarios, accumulating experience. After several episodes of exploration and updates to the network parameters, the reward curve begins to rise sharply, indicating that the policy has become proficient and can effectively handle the problem, and the agent continues to adapt to different scenarios. At approximately 100,000 steps, the reward curve peaks and then fluctuates with minimal volatility over a longer period. This indicates that the optimal policy has been learned at approximately 100,000 steps, and the subsequent fluctuations are due to the randomness of traffic scenarios and traffic flow at different times.

[0118] like Figure 7 , Figure 8 The diagram illustrates the changes in the total number of vehicles and the average waiting time. The trends of these two reward terms are largely consistent with the reward curve. In the initial few thousand steps, the total number of vehicles increases, then decreases rapidly with policy updates, reaching a low point around 100,000 steps, and subsequently fluctuating within a small range. The average waiting time reaches its lowest point around 75,000 steps after the policy update and remains relatively stable thereafter with only slight fluctuations. Although these reward terms converge at different points in time, the trends and the final result are consistent. This indicates that the convergence of rewards is accompanied by the convergence of reward terms, and that under the influence of the Intersection-Transformer algorithm, the traffic lights at the intersection become more efficient in managing traffic flow, thereby improving the traffic conditions at the intersection.

[0119] Experimental results demonstrate that the Intersection-Transformer algorithm not only optimizes traffic signal control strategies but also significantly improves the overall operational efficiency of the traffic system. By reducing average vehicle waiting time and adjusting vehicle flow distribution, the algorithm effectively enhances intersection capacity, which is particularly important for urban traffic management. This improvement in strategy and optimization of the algorithm ultimately leads to smoother traffic flow and higher traffic system efficiency.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive traffic signal control method based on reinforcement learning and self-attention mechanisms, characterized in that, The method includes: A Markov decision process model and a policy neural network are established based on a deep reinforcement learning algorithm. The observation space constraining the Markov decision process is determined based on the sum of various traffic flow information at the intersection. The phase of the traffic lights at the intersection constrains the behavior space of the Markov decision process. The reward function of the Markov decision process is constrained based on the weighted sum of the total number of vehicles, speed, queue length, and waiting time. A deep Q-network and self-attention mechanism were built based on the Markov decision process model; The policy is evaluated and the neural networks of the policy neural network, encoder, and decoder are updated; The neural network is trained iteratively multiple times to obtain the trained neural network model; Based on the trained neural network model, real-time traffic signal control signals are generated. The observation space is o= The observed subset is ,in Indicates time Lane The total number of vehicles in the country Indicates time Lane Medium speed below The average waiting time for vehicles Indicates that in the lane The waiting time for vehicle m; Indicates time Lane queue length, Indicates time Lane The average speed of vehicles in the middle, Indicates that in the lane The speed of vehicle m in the middle. This indicates the maximum permitted speed for that lane; Action space is The intelligent agent at time step After observation Execute actions That is, a traffic signal phase ,implement The intelligent agent decides Lanes that are permitted to pass and lanes that are not permitted to pass at any time, as well as phase maintenance Time; The reward function consists of the following parts: 、 、 、 ； ； in This indicates the average waiting time for vehicles. This indicates the average queue length of vehicles. This represents the average speed of the vehicle. Indicates the total number of vehicles. Indicates the total number of lanes. These are the weighting coefficients for the four reward sub-items; The steps of evaluating the policy and updating the neural networks of the policy neural network, encoder, and decoder specifically include: Sample the agent's trajectory; The current strategy is evaluated based on the reward function and the value function; The neural network parameters of the policy network, encoder, and decoder are optimized and updated based on the proximal policy. The specific steps for evaluating the current strategy based on the reward function and the value function include: Calculate the reward function; Calculate the state value function and the behavior value function based on the reward function and the observed probability distribution; Calculate the advantage function based on the state value function and the behavior value function; The current strategy is evaluated based on the advantage function results; The encoding process of the encoder specifically includes: The observations are processed through an embedding layer; The processed data is then connected via a self-attention mechanism and residual connections. The observation information is further processed by a multilayer perceptron to obtain a representation of the observation information; The decoding process of the decoder specifically includes: The initial action flag is processed through an embedding layer; The processed data is then processed through the first self-attention mechanism; The data and observation information obtained from the first self-attention mechanism are processed by the second self-attention mechanism and the multilayer perceptron to obtain the action representation.

Images