Adaptive traffic signal control method based on reinforcement learning and self-attention mechanism

By combining reinforcement learning and self-attention mechanisms in adaptive traffic signal control, using Transformer model and PPO optimized signal control strategy, the problems of instability in training and high computing resources in complex traffic scenarios are solved, and the problem of instability in training in complex traffic scenarios is realized. More efficient traffic signal control significantly improves the efficiency of the traffic system.

CN118942261AActive Publication Date: 2024-11-12JILIN UNIVERSITY

Patent Information

Application Number
CN202411089512.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-12
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The existing adaptive traffic signal control algorithms are difficult to capture dynamic changes, high-dimensional information characteristics and insufficient generalization capabilities when dealing with complex traffic scenarios, resulting in unstable training and high consumption of computing resources, making it difficult to effectively apply in new environments.

Method used

Adaptive traffic signal control method based on reinforcement learning and self-attention mechanism is adopted, Markov decision-making process model is established through deep reinforcement learning algorithm, combined with Transformer model and introduced Proximal Policy Optimization (PPO) to use the total number of vehicles, speed and queue length and wait time are reward functions, optimize signal control strategies, improve model generalization ability and training stability.

Benefits of technology

Effectively reduces training time, improves the processing capacity of complex traffic scenarios, shortens vehicle waiting time, reduces queue length, and significantly improves the efficiency of the traffic system, especially in complex traffic conditions, which is superior in vehicle waiting time compared to traditional methods Shorten by more than 40%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942261A_ABST
    Figure CN118942261A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of traffic signal control, and particularly discloses a self-adaptive traffic signal control method based on reinforcement learning and a self-attention mechanism, and the method comprises the steps: building a Markov decision process model and a strategy neural network based on a deep reinforcement learning algorithm; the method comprises the following steps: establishing a deep Q network and a self-attention mechanism based on a Markov decision process model, evaluating a strategy, updating a strategy neural network and neural networks of an encoder and a decoder, carrying out iterative training on the neural networks for multiple times to obtain a trained neural network model, and according to the trained neural network model, obtaining a self-attention mechanism. According to the algorithm, real-time traffic signal control signals are generated, optimal strategies can be learned step by step in complex traffic flows, in addition, the algorithm adopts an encoder-decoder structure, historical tracks are stored in a playback buffer, and parameters are updated through near-end strategy optimization. The problem that an existing adaptive traffic signal control method needs a large amount of data and a large amount of computing resources due to training is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of traffic signal control, and in particular to an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism. Background Art

[0002] In recent years, due to population growth and accelerated urbanization, urban travel demand has surged. Traffic congestion is the product of the imbalance between traffic supply (road capacity) and travel demand. As a key node in the urban road network, intersections will gradually become a bottleneck in the urban road network capacity, resulting in the inability to continuously improve traffic efficiency.

[0003] In the past, in order to meet the growing traffic demand in cities, the supply capacity of urban roads was improved by building new road traffic infrastructure, and the growth of traffic demand was met from the perspective of urban traffic supply. However, due to the limitation of funds and road space, it is not an economical and effective long-term strategy. Therefore, under the condition of limited road supply, the focus of research should be shifted to organizing and optimizing the existing travel demand from the perspective of urban traffic demand through traffic management and control measures. As the core of urban traffic management and control, traffic signal control can maximize the traffic volume of intersections, improve the operation efficiency of urban road networks and the traffic capacity of intersections, reduce the frequency and intensity of various traffic conflicts, and thus alleviate urban traffic congestion.

[0004] Currently, many intersections use fixed-time traffic signal systems that set signal durations based on historical data rather than adjusting them based on real-time traffic demand. Traditional traffic signal control techniques usually use predefined rules derived from expert experience-based strategies. These methods lack the ability to dynamically adjust signal phases based on instantaneous traffic flow data. Adaptive traffic signal control is particularly important in this context. Adaptive Traffic Signal Control (ATSC) is a hot topic in the study of intelligent transportation systems. It can optimize traffic flow in regional road networks and effectively reduce congestion by dynamically changing signal phases to respond to current traffic conditions. Although adaptive traffic signal control can alleviate urban traffic congestion to a certain extent, it is mainly based on predictions of fixed traffic models, selection of preset signal control schemes, or real-time prediction of traffic demand based on traffic simulation models, such as Split, Cycle and Offset Optimisation Technique (SCOOT) and Sydney Coordinated Adaptive Traffic System (SCATS). They can adjust traffic lights according to real-time traffic volume, but they are essentially driven by traffic models. They need to calibrate traffic simulation model parameters and design predefined signal control schemes according to actual traffic scenarios in advance. Due to the dynamic, random and uncertain characteristics of the actual traffic environment, the applicability of simulation model parameters and preset signal control schemes is poor.

[0005] Although traditional reinforcement learning-based ATSC algorithms are theoretically believed to have the ability to adjust traffic signals, they face several limitations in practical applications. First, these algorithms often have difficulty accurately capturing the correlation between time series and fully understanding complex traffic scenarios when dealing with the dynamic changes in traffic flow and the high-dimensional nature of information. This lack of deep insight causes significant non-stationarity and instability in the training process. Second, most ATSC algorithms need to consume a large amount of data to achieve efficient policy learning, which not only increases the dependence on computing resources but also significantly prolongs the learning process. Third, these algorithms are usually optimized in a specific training environment, making them difficult to generalize and transfer when facing new or unknown environments, limiting their application in a wider range of complex traffic environments. Due to these factors, RL algorithms are still feasible in dealing with simple scenarios, but their effectiveness and adaptability in complex real-world traffic conditions are greatly limited. Summary of the invention

[0006] The purpose of the present invention is to provide an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism, the method comprising: Based on the deep reinforcement learning algorithm, a Markov decision process model and a policy neural network are established. The observation space of the Markov decision process is determined based on the sum of various traffic flow information at the intersection. The phase of the traffic lights at the intersection constrains the behavior space of the Markov decision process. The reward function of the Markov decision process is constrained by the weighted sum of the total number of vehicles, speed, queue length, and waiting time. Build a deep Q network and self-attention mechanism based on the Markov decision process model; Evaluate the policy and update the policy neural network, encoder and decoder neural networks; Perform multiple iterations of training on the neural network to obtain a trained neural network model; Generate real-time traffic signal control signals based on the trained neural network model.

[0008] As a further solution of the present invention, the steps of evaluating the strategy and updating the strategy neural network, the encoder and the decoder neural network specifically include: Sampling agent trajectories; Evaluate the current strategy based on the reward function and value function; The policy network and the neural network parameters of the encoder and decoder are updated according to the proximal policy optimization.

[0009] As a further solution of the present invention, the step of evaluating the current strategy according to the reward function and the value function specifically includes: Calculate the reward function; Calculate the state value function and behavior value function based on the reward function and the observed probability distribution; Calculate the advantage function based on the state value function and the behavior value function; Evaluate the current strategy based on the advantage function results.

[0010] As a further solution of the present invention, the encoding process of the encoder specifically includes: Process the observations through an embedding layer; The processed data is passed through the self-attention mechanism and residual connection; The observed information is further processed through a multi-layer perceptron to obtain a representation.

[0011] As a further solution of the present invention, the decoding process of the decoder specifically includes: The starting action flag is processed through the embedding layer; The processed data is processed through the first self-attention mechanism; The representation of the data and observation information obtained by the first self-attention mechanism is processed by the second self-attention mechanism and the multi-layer perceptron to obtain the action representation.

[0012] As a further solution of the present invention, the observation space is o= , the observation subset is ,in Indicates at time Lane The total number of vehicles in Indicates at time Lane Medium speed below The average waiting time of vehicles, Indicates in the lane The waiting time of vehicle m in Indicates at time Lane The queue length, Indicates at time Lane The average speed of vehicles in Indicates in the lane The speed of vehicle m in Indicates the maximum speed allowed in the lane; The action space is , the agent at time step After obtaining the observation , perform the action , i.e. a traffic signal phase ,implement The agent decides Lanes that are always allowed and not allowed to pass, and phase maintenance time.

[0013] As a further solution of the present invention, the reward function consists of the following parts: , , , ; ; in represents the average waiting time of vehicles, represents the average queue length of vehicles, represents the average speed of the vehicle, represents the total number of vehicles, represents the total number of lanes, They are the weight coefficients of the four reward sub-items respectively.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Intersection-Transformer utilizes Transformer, which can process data in parallel, thus helping to reduce training time. In addition, the efficient computational properties of the algorithm help to handle larger state and action spaces. The self-attention mechanism of Intersection-Transformer is able to capture dynamic changes in the environment, solving the problems of non-stationarity and training instability. It can also capture the details of general feature representations, which may improve the generalization ability of the model in new environments. We also introduced PPO, which uses truncated probability ratios to limit the policy update step, preventing large policy changes during training and increasing the stability of the training process.

[0015] The ATSC problem is modeled in a refined manner using the Markov Decision Process (MDP), and relevant parameters such as the observation space, action space, and reward function are carefully designed, effectively solving the problems of long training time and low training efficiency of other ATSC methods.

[0016] Extensive experiments were conducted in various traffic scenarios. Experimental results show that our IT consistently performs well in alleviating traffic congestion and improving the efficiency of traffic systems. As the complexity of intersections increases, the advantages of the Intersection-Transformer algorithm gradually emerge. Under the most complex traffic conditions, the ability of the public method to handle traffic flow, such as shortening vehicle waiting time and reducing intersection queue length, is more than 40% higher than the traditional reinforcement learning-based ATSC algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0018] Figure 1 A flowchart of an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism provided in an embodiment of the present invention.

[0019] Figure 2 An architecture diagram of an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism provided in an embodiment of the present invention.

[0020] Figure 3 A flowchart of the steps of evaluating a strategy and updating the strategy neural network, the encoder and the decoder neural network provided in an embodiment of the present invention.

[0021] Figure 4 An architectural diagram of the steps for evaluating a strategy and updating the strategy neural network, the encoder neural network, and the decoder neural network provided by an embodiment of the present invention.

[0022] Figure 5 A flowchart of evaluating the current strategy steps according to the reward function and the value function provided in an embodiment of the present invention.

[0023] Figure 6 A reward performance curve diagram provided by an embodiment of the present invention.

[0024] Figure 7 A curve chart of reward sub-items (total number of vehicles) provided in an embodiment of the present invention.

[0025] Figure 8 A curve chart of reward sub-items (average waiting time) provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] like Figure 1 , Figure 2 As shown, in an embodiment of the present invention, an adaptive traffic signal control method based on reinforcement learning and self-attention mechanism, the method specifically includes: Step S100, establishing a Markov decision process model and a policy neural network based on a deep reinforcement learning algorithm, and determining the observation space of the Markov decision process based on the sum of various traffic flow information at the intersection, constraining the behavior space of the Markov decision process by the phase of the traffic lights at the intersection, and constraining the reward function of the Markov decision process based on the weighted sum of the total number of vehicles, speed, queue length, and waiting time; Step S200, building a deep Q network and a self-attention mechanism based on a Markov decision process model; Step S300, evaluating the strategy and updating the strategy neural network, the encoder neural network and the decoder neural network; Step S400, performing multiple iterations of training on the neural network to obtain a trained neural network model; Step S500, generating a real-time traffic signal control signal according to the trained neural network model.

[0028] The present invention discloses an adaptive traffic signal control reinforcement learning algorithm named Intersection-Transformer, and applies it to the vehicle cooperative control of intersections. The algorithm is integrated with Transformer, and according to traffic information such as traffic density and vehicle waiting time, comprehensively considers factors such as fairness and efficiency, so that the intersection signal lights can scientifically and reasonably adjust the phase and duration adaptively, thereby achieving efficient traffic and reducing the occurrence of traffic congestion. The Intersection-Transformer algorithm introduces the Transformer model on the basis of reinforcement learning to ensure that the traffic signal control agent can capture the potential connection between different traffic information. The algorithm collects information related to traffic conditions in the environment and interacts with the environment according to the actions generated by the policy neural network; at the same time, the strategy is evaluated through the reward function and the value neural network, and the proximal policy optimization (PPO) is used to update the policy network and the value neural network parameters, which solves the problem that the existing adaptive traffic signal control method requires a large amount of data and a large amount of computing resources for training. The present invention aims to improve the operating efficiency and traffic capacity of urban intersections through scientific and reasonable signal control schemes, maximize the traffic volume of intersections, and effectively alleviate urban traffic congestion. The present invention applies the powerful understanding and reasoning capabilities of LLM to ATSC to adapt to the high dynamics of traffic conditions and the high dimensionality of traffic information. This enables Intersection-Transformer to gradually learn the optimal strategy in complex traffic flows. In addition, this algorithm adopts an encoder-decoder structure, stores historical trajectories in an experience replay buffer, and uses proximal policy optimization (PPO) to update parameters.

[0029] The agent interacts with the environment at the intersection and observes the current environment in real time. The current strategy collected by the Intersection-Transformer algorithm The observation space provided by the environment is The policy neural network maps the state space to the behavior space , and select actions from the behavior space for the agent to execute. The agent interacts with the environment based on the actions obtained, obtains a new observation space and reward function, and thus forms a closed loop of interaction between the algorithm and the environment. When the number of strategy updates reaches a certain level, the training ends.

[0030] As a preferred embodiment of the present invention, the observation space is used to describe the observation information when the intelligent agent interacts with the environment. A reasonable observation space is crucial for the effective training of deep reinforcement learning algorithms. In the ATSC system, it is very important to be able to accurately extract and reconstruct the observation information in the complex dynamic environment of the intersection, because this determines the ability to accurately output appropriate actions.

[0031] The observation space is defined as the sum of observations of lanes in the current scene, i.e., o = Considering that the traffic conditions at the intersection vary based on the dynamic and static traffic information of each lane, the observation subset is set to ,in Indicates at time Lane The total number of vehicles in; Indicates at time Lane Medium speed below The average waiting time of vehicles, Indicates in the lane The waiting time of vehicle m in ; Indicates at time Lane The queue length can be represented by the number of queued vehicles; Indicates at time Lane The average speed of vehicles in Indicates in the lane The speed of vehicle m in Indicates the maximum speed allowed for the lane. The above observation space can adapt to complex and randomly changing traffic intersection scenes and effectively characterize the traffic conditions at the intersection.

[0032] Consider the set of discrete phases of a traffic light as the agent’s action space The agent at time step After obtaining the observation , perform the action , i.e. a traffic signal phase ,implement The intelligent experience determines Lanes that are always allowed and not allowed to pass, and phase maintenance time.

[0033] As a preferred embodiment of the present invention, the reward function is the quantitative feedback received by the agent from the environment after performing an action. In this study, the reward function will guide the agent to continuously explore and learn the optimal strategy while maximizing the reward. The design of these functions has a key impact on the progressive performance and convergence speed of the deep reinforcement learning algorithm. If the reward function over-optimizes a specific scenario, it may lead to overfitting problems, which reduces the ability of the trained model to adapt to new scenarios. The general rule is that performance indicators should be designed based on the goals that are actually hoped to be achieved in the environment, rather than based on how people think the agent should behave. In order to fully describe the environment, the reward function focuses on comprehensively improving the traffic efficiency of the intersection, ensuring that the reward system can truly reflect the desired results in the environment, rather than just reflecting how the agent should behave. Such a design helps to avoid the problem of over-training in a specific environment, while enhancing the adaptability and versatility of the model in new scenarios. Therefore, taking all factors into consideration, the reward function consists of the following parts ( Indicates the total number of lanes): Total number of vehicles ( ): ; By focusing on the total number of vehicles, the reward function can directly affect and evaluate the traffic flow at the intersection. Reducing the total number of vehicles at the intersection can help alleviate traffic congestion and make traffic flow smoother. This indicator can reflect the traffic capacity and flow management efficiency of the intersection; The average speed of the vehicle ( ): ; Average speed is one of the key indicators for evaluating traffic efficiency. By increasing the average speed of vehicles, traffic congestion can be reduced, travel time can be shortened, and the traffic efficiency of the entire intersection can be improved. A higher average speed usually means good traffic conditions and vehicles do not need to stop or slow down frequently; The average queue length of vehicles ( ): ; This indicator reflects the vehicle queue status at the intersection. Short queue length means that vehicles can pass through the intersection faster, reducing waiting time and dwell time, which helps to improve the continuity of traffic flow and reduce traffic congestion. Controlling queue length can effectively optimize the traffic flow distribution and signal light scheduling at the intersection; The average waiting time of vehicles ( ): ; Average waiting time is an important indicator for measuring intersection efficiency. By reducing the average waiting time of vehicles at intersections, the smoothness and efficiency of traffic flow can be significantly improved. Lower waiting time not only reduces driver inconvenience and anxiety, but also helps reduce vehicle fuel consumption and emissions.

[0034] In summary, the reward function is finally designed as: ; in They are the weight coefficients of the four reward sub-items respectively.

[0035] The final goal of the proposed algorithm is the weighted sum of these four objectives. This approach can be considered as a normalization method. This weighted summary method is chosen because at the initial stage of each environment, the scales of these four factors are different. If they are simply added up as a reward, it may cause the agent to only focus on the indicators with larger values, resulting in the algorithm failing to converge.

[0036] By weighting these four indicators, we can ensure that each indicator has an appropriate proportion in the reward function, thus overcoming the problem of different dimensions and uneven influence of each indicator. This design helps the algorithm balance the importance of different indicators and avoids over-optimizing a single indicator while ignoring other equally important indicators. In addition, this weighting method also helps to improve the stability and convergence of the algorithm, so that the algorithm can effectively learn and adapt in a variety of traffic environments, ultimately achieving the goal of reducing congestion and improving traffic efficiency.

[0037] The policy neural network receives observations from the environment As input to the neural network, it outputs the actions of the agent in the intersection environment. and its probability . Compute the value of the input observation in the process of outputting the agent's action The agent in time Take action when The new state observed later is , the agent will also receive environmental rewards If the length of the trajectory is , the trajectory used by the Intersection-Transformer algorithm is .

[0038] like Figure 3 , Figure 4 , Figure 5 As shown, as a preferred embodiment of the present invention, the steps of evaluating the strategy and updating the strategy neural network, the encoder and the decoder neural network specifically include: Step S301, sampling the agent trajectory; Step S302, evaluating the current strategy according to the reward function and the value function; Step S303, optimizing and updating the neural network parameters of the strategy network and the encoder and decoder according to the proximal strategy.

[0039] The observation space is used to describe the observation information when the agent interacts with the environment. A reasonable observation space is crucial for the effective training of deep reinforcement learning algorithms. In the ATSC system, it is very important to accurately extract and reconstruct the observation information in the complex dynamic environment of the intersection, because this determines the ability to accurately output appropriate actions. This substep collects the sequence of observed behaviors of the agent, and the policy neural network receives the observations transmitted by the environment. As input to the neural network, it outputs the actions of the agent in the intersection environment. and its probability . Compute the value of the input observation in the process of outputting the agent's action The agent in time Take action when The new state observed later is , the agent will also receive environmental rewards If the length of the trajectory is , the trajectory used by the Intersection-Transformer algorithm is .

[0040] Based on the collected observed behavior trajectories of the intelligent agent, the strategy is evaluated and optimized. The reward function and value function are used to evaluate the performance of the current strategy. The reward function is used to evaluate the degree to which the strategy achieves the expected cumulative reward of the intelligent agent, and the neural network parameters of the strategy network, encoder, and decoder are updated through proximal strategy optimization.

[0041] As a preferred embodiment of the present invention, the step of evaluating the current strategy according to the reward function and the value function specifically includes: Step 3021, calculating the reward function; Step 3022, calculating the state value function and the behavior value function according to the reward function and the observed probability distribution; Step 3023, calculating the advantage function according to the state value function and the behavior value function; Step 3024, evaluate the current strategy based on the advantage function result.

[0042] As a preferred embodiment of the present invention, the encoding process of the encoder specifically includes: Process the observations through an embedding layer; The processed data is passed through the self-attention mechanism and residual connection; The observed information is further processed through a multi-layer perceptron to obtain a representation.

[0043] As a preferred embodiment of the present invention, the decoding process of the decoder specifically includes: The starting action flag is processed through the embedding layer; The processed data is processed through the first self-attention mechanism; The representation of the data and observation information obtained by the first self-attention mechanism is processed by the second self-attention mechanism and the multi-layer perceptron to obtain the action representation.

[0044] The conceptual structure of Intersection-Transformer mainly consists of two parts: Encoder and Decoder. The encoder is responsible for receiving observations from the environment. , and learn the representation of the observations The decoder is responsible for outputting the actions that the agent will perform. . Therefore, after the agent executes a predetermined number of steps, the partial history Will be provided to the PPO for policy renewal and encoder-decoder networks.

[0045] Advantage Function is used in strategy evaluation and optimization, denoted as , is used to evaluate the relative advantage of taking a specific action in a given state compared to the average situation. is the behavior value function, which represents the expected return of taking an action in the state and following a certain strategy; and is the state value function, which represents the expected return based only on the state. The calculation formulas are: ; ; ; in, Indicates observation The probability distribution of represents the discounted reward function .

[0046] The core idea of ​​the advantage function is to provide a criterion for the reinforcement learning algorithm to measure the difference between taking a specific action and the average level of the execution strategy. This concept is particularly important in methods such as policy gradient methods because it can help us adjust the policy parameters more accurately and thus optimize the learning process. Through the advantage function, the learning algorithm can identify and prioritize actions that can bring higher additional rewards than the average performance.

[0047] The main function of the encoder is to extract useful features from the raw environment input and transform these features into more abstract and decision-making internal representations. The parameters of the encoder are calculated using It indicates that its function is as follows: First, through an embedding layer , whose function is to map discrete inputs into continuous digital vectors in low-dimensional space to improve computational efficiency. Then, through the self-attention mechanism and residual connection To prevent gradient disappearance and network degradation. Finally, the data is processed by a multi-layer perceptron (MLP) to obtain , which is one of the inputs to the decoder. In addition, Also used to calculate the value , which is convenient for subsequent updates within the network. The role of the encoder is similar to that of a judge, so our goal is to make it approximate the value function, which aims to minimize the empirical Bellman error in the following way: ; in, represents the loss function, represents the parameters of the target network, which is non-differentiable and is updated every few time steps. represents the number of updates of the encoder parameters during training, represents the reward function, Represents the discount factor. , Represent the value function of the target network and the current value function respectively.

[0048] The task of the decoder is to convert the representation learned by the encoder into specific action instructions. This is usually achieved through a series of neural network layers that generate the final action output based on the learned features and possible policy guidelines. The design of the decoder ensures that the output actions are adapted to the needs of the specific task and also reflect the decision preferences of the policy network in a specific environment state.

[0049] Embedded Actions (an arbitrary symbol indicating the start of decoding) is passed to the decoder, whose parameters are given by The decoder consists of many decoder blocks similar to the encoder. Each block contains two attention mechanisms and a multi-layer perceptron. The first mechanism is responsible for receiving , and generate ,in is obtained after dimension transformation through the embedding layer. In the second attention mechanism, the attention score is calculated as After passing through two MLPs, the action representation The second MLP outputs the probability distribution of the agent's actions, i.e., the policy Next, we try to minimize the Clip PPO objective for training the decoder: ; ; in, represents the target loss function. It is a ratio that indicates the relative probability between the new strategy and the old strategy after the update. is an estimate of the advantage function.

[0050] In the update process of PPO, the use of historical trajectories τ is crucial for evaluating and adjusting the current strategy. By replaying these historical data, PPO can evaluate the effectiveness of the existing strategy and optimize the network parameters of the encoder and decoder accordingly, thereby achieving more accurate and effective control in the continuous decision-making process. This trajectory-based learning method helps capture time dependencies in complex environments, thereby improving the overall policy performance.

[0051] This patent simulates and verifies the disclosed method. The simulation platform is SUMO, and a reinforcement learning model is built based on the PyTorch framework. The intersection in the simulation experiment of this patent has a total of 32 lanes, including 3 through lanes, 1 right turn lane and 1 left turn lane in each direction. This layout helps to effectively handle traffic flows in all directions. The length of all lanes is 750 meters. This design can accommodate a large number of vehicles, especially during peak traffic hours, which helps to alleviate traffic congestion. During the training process, the traffic flow density at the intersection was approximately 4,000 vehicles / hour. This density represents the typical traffic flow during peak hours in the city, providing a challenging environment for the algorithm to optimize and test its performance. This high-traffic environment simulates actual urban traffic conditions, making the trained model more suitable for complex traffic management tasks in the real world. In the SUMO environment, the shortest simulation time interval (i.e., time step) is ) is 1 second. Within 3600 seconds, the randomly generated traffic flow will enter the incoming lane and then pass through the intersection. The Intersection-Transformer algorithm has been trained 3 million times in this training scenario, with each episode length of 500 seconds, which means a total of 6000 episodes. In this virtual simulation scenario, the weights of each reward sub-item in the discounted reward function are Every decision The interval is 6 seconds, which means that the agent decides every 6 seconds whether to keep the current phase or switch to another phase. If the time is set too short, such as 1 second, the traffic lights may switch too frequently, which is impractical and increases the computational burden. If it is set too long, it will reduce the learning rate of the agent. For detailed parameters of the SUMO simulator, Intersection-Transformer algorithm, and Deep Q Network (DQN) algorithm, see Table 1 for experimental parameter settings.

[0052]

[0053] In the performance comparison experiment, we deeply explore the performance comparison between the strategy trained by the Intersection-Transformer algorithm and the DQN algorithm in actual traffic management. In order to comprehensively evaluate the effect of the Intersection-Transformer algorithm and demonstrate its applicability under different traffic conditions, we carefully designed two typical test scenarios: Simple scenario: This scenario is designed to be a traffic flow of lower complexity, mainly simulating one-way flow from north to south and from west to east. Each direction includes two lanes, and the overall traffic density is set to 600 vehicles per hour. This setting is designed to test the performance of the algorithm under low-stress conditions, where the green light phase is simply switched. This scenario reflects low-volume traffic conditions in the city, such as residential areas or roads during non-peak hours.

[0054] Difficult scenario: Similar to the training environment, this scenario simulates higher traffic density and more complex traffic dynamics. There are four lanes in each direction, and the traffic density is as high as 4,000 vehicles per hour. In addition, the green light phase setting is more complex and contains multiple modes. This scenario is designed to simulate the conditions of rush hour in the city or on major traffic arteries, challenging the performance of the algorithm in a high-stress and high-complexity environment.

[0055] In the above scenario, several experiments are also set up where the green light phase is fixed in order to compare with the Intersection-Transformer algorithm: FIXED-6 / -12 / -18 / -24: These numbers represent the fixed duration of each green phase. In the FIXED experiment, the green phases will cycle with fixed durations to affect traffic conditions, similar to the traffic lights we see at most intersections in real life.

[0056]

[0057] Table 2 shows the final results of the performance comparison between the fixed green phase duration, Intersection-Transformer algorithm, and DQN algorithm in different traffic scenarios. In simple scenarios, due to the relatively low traffic flow, the performance difference between the fixed green phase and the adaptive green phase is not significant. This shows that in an environment with simpler and more predictable traffic, even a simple fixed-phase control can effectively manage traffic flow.

[0058] However, as the size of intersections increases and traffic flows increase, the superiority of the Intersection-Transformer algorithm gradually becomes apparent. In complex traffic scenarios, the Intersection-Transformer algorithm shows significant performance improvements compared to DQN and fixed-phase control. Specifically, in terms of the number of vehicles and vehicle passing speed, the performance of the Intersection-Transformer algorithm has achieved a significant improvement of about 50%. This improvement not only reduces traffic pressure, but also improves the efficiency of traffic flow, significantly optimizing vehicle capacity and road utilization.

[0059] The Intersection-Transformer algorithm also performed well in terms of vehicle queue length, reducing the queue length by about 30%. This improvement in indicators is of great significance for alleviating traffic congestion and reducing vehicle waiting time, especially during peak traffic hours.

[0060] like Figure 6 As shown in the figure, the curve represents the discounted reward that changes with the increase of time steps. From the initial state to the first few thousand steps, the agent explores the impact of various behaviors in different scenarios and accumulates experience. After several episodes of exploration and updating of network parameters, the reward curve begins to rise sharply, indicating that the strategy has become proficient in effectively handling the problem and the agent continues to adapt to different scenarios. At about 100,000 steps, the reward curve peaks and then fluctuates with minimal fluctuations for a long time. This shows that the optimal strategy has been learned at about the 100,000 step mark, and the subsequent fluctuations are caused by the randomness of traffic scenes and traffic flows at different times.

[0061] like Figure 7 , Figure 8As shown, the changes in the total number of vehicles and the average waiting time of vehicles are shown. The trends of these two reward items are basically consistent with the reward curve. In the first few thousand steps, the total number of vehicles increased, and then decreased rapidly with the update of the policy, reaching a low point at about 100,000 steps, and then fluctuated within a smaller range. The average waiting time of the vehicle reached its lowest point at about 75,000 steps after the policy was updated, and remained relatively stable thereafter, with only slight fluctuations. Although these reward items converge at different time points, the trends and final results are consistent. This shows that the convergence of the reward is accompanied by the convergence of the reward item, and under the influence of the Intersection-Transformer algorithm, the traffic lights at the intersection have become more efficient in managing traffic flow, thereby improving the traffic conditions at the intersection.

[0062] The experimental results show that the Intersection-Transformer algorithm not only optimizes the traffic signal control strategy, but also significantly improves the overall operating efficiency of the traffic system. By reducing the average waiting time of vehicles and adjusting the distribution of vehicle flow, the algorithm effectively improves the traffic capacity of intersections, which is particularly important for urban traffic management. This strategy improvement and algorithm optimization ultimately lead to smoother traffic flow and higher traffic system efficiency.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive traffic signal control method based on reinforcement learning and self-attention mechanism, characterized in that: The method comprises: Based on the deep reinforcement learning algorithm, a Markov decision process model and a policy neural network are established. The observation space of the Markov decision process is determined based on the sum of various traffic flow information at the intersection. The phase of the traffic lights at the intersection constrains the behavior space of the Markov decision process. The reward function of the Markov decision process is constrained by the weighted sum of the total number of vehicles, speed, queue length, and waiting time. Build a deep Q network and self-attention mechanism based on the Markov decision process model; Evaluate the policy and update the policy neural network, encoder and decoder neural networks; Perform multiple iterations of training on the neural network to obtain a trained neural network model; Generate real-time traffic signal control signals based on the trained neural network model; The observation space is o= , the observation subset is ,in Indicates at time Lane The total number of vehicles in Indicates at time Lane Medium speed below The average waiting time of vehicles, Indicates in the lane The waiting time of vehicle m in Indicates at time Lane The queue length, Indicates at time Lane The average speed of vehicles in Indicates in the lane The speed of vehicle m in Indicates the maximum speed allowed in the lane; The action space is , the agent at time step After obtaining the observation , perform the action , i.e. a traffic signal phase ,implement The agent decides Lanes that are always allowed and not allowed to pass, and phase maintenance time; The reward function consists of the following parts: 、 、 、 ; ; in represents the average waiting time of vehicles, represents the average queue length of vehicles, represents the average speed of the vehicle, represents the total number of vehicles, represents the total number of lanes, They are the weight coefficients of the four reward sub-items respectively.

2. The adaptive traffic signal control method based on reinforcement learning and self-attention mechanism according to claim 1 is characterized in that: The steps of evaluating the strategy and updating the strategy neural network, the encoder neural network and the decoder neural network specifically include: Sampling agent trajectories; Evaluate the current strategy based on the reward function and value function; The policy network and the neural network parameters of the encoder and decoder are updated according to the proximal policy optimization.

3. The adaptive traffic signal control method based on reinforcement learning and self-attention mechanism according to claim 2 is characterized in that: The step of evaluating the current strategy according to the reward function and the value function specifically includes: Calculate the reward function; Calculate the state value function and behavior value function based on the reward function and the observed probability distribution; Calculate the advantage function based on the state value function and the behavior value function; Evaluate the current strategy based on the advantage function results.

4. The adaptive traffic signal control method based on reinforcement learning and self-attention mechanism according to claim 1 is characterized in that: The encoding process of the encoder specifically includes: Process the observations through an embedding layer; The processed data is passed through the self-attention mechanism and residual connection; The observed information is further processed through a multi-layer perceptron to obtain a representation.

5. The adaptive traffic signal control method based on reinforcement learning and self-attention mechanism according to claim 4 is characterized in that: The decoding process of the decoder specifically includes: The starting action flag is processed through the embedding layer; The processed data is processed through the first self-attention mechanism; The representation of the data and observation information obtained by the first self-attention mechanism is processed by the second self-attention mechanism and the multi-layer perceptron to obtain the action representation.

Citation Information

Patent Citations

  • Self-adaption traffic signal control system and method based on deep reinforcement learning

    CN110047278A

  • Deep reinforcement learning traffic signal control method based on self-attention mechanism

    CN115762128A

  • Signal control apparatus and signal control method based on reinforcement learning

    US20220076571A1

Cited By

  • Traffic signal lamp and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning

    CN119600826A

  • Emergency vehicle priority passing control method based on parameterized reinforcement learning

    CN119832755A

  • Route linkage type traffic signal control method and system based on reinforcement learning

    CN119851491A

  • Photovoltaic power station parameter identification method based on Transform-PPO deep reinforcement learning

    CN119940134A

  • Traffic signal lamp induction control method and system based on deep reinforcement learning

    CN120279738A