Traffic signal control method, device and system and storage medium
By combining the multi-agent advantage decomposition theorem and the heterogeneous agent mirror learning framework with the Transformer model, the traffic signal control model is optimized, solving the problems of credit allocation and policy instability in multi-agent systems. This achieves efficient coordinated control of traffic signals at multiple intersections, improving both traffic efficiency and learning efficiency.
Patent Information
- Application Number
- CN202410975629.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing multi-agent reinforcement learning methods in traffic signal control suffer from problems such as unclear credit allocation, policy instability, and convergence issues, resulting in unsatisfactory traffic signal control performance.
By employing the multi-agent advantage decomposition theorem and the heterogeneous agent mirror learning framework, combined with the Transformer model, the MATLight algorithm is designed to optimize the traffic signal control model through sequential decision-making and the heterogeneous agent mirror learning framework, thereby achieving coordinated control of traffic signals at multiple intersections.
It improves the coordination of traffic signal control, enhances overall traffic efficiency, reduces traffic congestion, and has faster convergence speed and higher learning efficiency.
Smart Images

Figure CN118918716A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of road traffic technology, and in particular to a traffic signal control method, device, system and storage medium. Background Art
[0002] With the acceleration of urbanization and the increase in travel demand, people have higher requirements for intelligent traffic signal control technology. How to solve the above difficulties and meet the needs of intelligent construction of urban management systems is urgent. The training and decision-making process of multi-agent reinforcement learning mainly faces problems such as credit allocation, strategy instability, complex interactions between agents, and low sample efficiency. These problems will lead to unsatisfactory traffic signal control effects in the algorithm. This unsatisfactory control effect may be caused by one of the problems or the combined effect of multiple problems.
[0003] With the continuous development of social economy and the accelerated construction of urbanization, traffic congestion has evolved into an increasingly severe and challenging social problem. The current traffic signal control system still faces some challenges in dealing with increasingly complex traffic conditions. Researchers have begun to pay attention to the difficulties of adaptive control in the field of traffic signal control, and the coordinated control of traffic lights at multiple intersections is also an important and challenging problem currently faced. In actual traffic scenarios, the traffic volume, road conditions and traffic demand at each intersection are often diverse and dynamically changing, which makes it difficult for traditional traffic signal control strategies to adapt to complex traffic environments and achieve good coordinated control.
[0004] As a self-learning decision-making algorithm, multi-agent reinforcement learning has demonstrated unparalleled advantages in traffic signal control tasks. However, existing methods still face problems in the decision-making process of multi-agent systems, such as the non-stationarity of algorithm training, the complex influence interaction between multiple agents that makes it difficult for the algorithm to learn and converge, and the low efficiency of sample learning. In the traffic signal control task, the key is how to clarify the contribution of traffic signal control at each intersection, and how multiple intersections influence and coordinate with each other, so as to improve the control effect of the entire area and alleviate overall traffic congestion. Summary of the invention
[0005] The present invention is provided to solve the above problems existing in the prior art. Therefore, a traffic signal control method, device, system and storage medium are needed, which combines the twin neural network with the similarity prediction idea, remodels the multi-intersection traffic signal control problem, improves the multi-agent reinforcement learning model algorithm, including the deep neural network structure, training mode, etc., optimizes the current traffic signal control method, selects the latest benchmark platform, and completes the intelligent control task of traffic signals on a public traffic simulator and data set.
[0006] According to a first embodiment of the present invention, a traffic signal control method is provided, the method comprising:
[0007] Determine the joint strategy based on the multi-agent advantage decomposition theorem;
[0008] Determining a heterogeneous agent mirror learning framework based on the joint strategy;
[0009] Construct a multi-intersection traffic signal control model, wherein the multi-intersection traffic signal control model includes a set of multi-intersection agents Joint Observation Space Joint Action Space Joint reward function R: Transition probability function P: and a discount factor γ∈[0,1), the joint observation space is the product of the local observation spaces of the intersection, the joint action space is the product of the local action spaces of the agents at the intersection, and the reward r of the i-th agent is i is the negative value of the pressure at the intersection. The pressure at the intersection is defined as the absolute value of the sum of the pressures in the traffic flow direction. The pressure at the i-th intersection is expressed as P i , the traffic signal controller at the i-th intersection is abstracted as the i-th agent in the set of multi-intersection agents, and the reward formula is expressed as r i =-P i ;
[0010] The multi-intersection traffic signal control model is solved using the heterogeneous agent mirror learning framework to obtain a final strategy.
[0011] According to a second technical solution of the present invention, a traffic signal control device is provided, the device comprising a processor, and the processor is configured to:
[0012] Determine the joint strategy based on the multi-agent advantage decomposition theorem;
[0013] Determining a heterogeneous agent mirror learning framework based on the joint strategy;
[0014] Construct a multi-intersection traffic signal control model, wherein the multi-intersection traffic signal control model includes a set of multi-intersection agents Joint Observation Space Joint Action Space Joint reward function R: Transition probability function P: and a discount factor γ∈[0,1), the joint observation space is the product of the local observation spaces of the intersection, the joint action space is the product of the local action spaces of the agents at the intersection, and the reward r of the i-th agent is i is the negative value of the pressure at the intersection. The pressure at the intersection is defined as the absolute value of the sum of the pressures in the traffic flow direction. The pressure at the i-th intersection is expressed as P i , the traffic signal controller at the i-th intersection is abstracted as the i-th agent in the set of multi-intersection agents, and the reward formula is expressed as r i =-P i ;
[0015] The multi-intersection traffic signal control model is solved using the heterogeneous agent mirror learning framework to obtain a final strategy.
[0016] According to a third technical solution of the present invention, there is provided a traffic signal control system, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method as described above.
[0017] According to a fourth technical solution of the present invention, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the method described above is executed.
[0018] The traffic signal control method, device, system and storage medium according to each solution of the present invention have at least the following technical effects:
[0019] 1) Model the coordinated control of traffic signals at multiple intersections as a multi-agent sequential decision-making process. Model the traffic signal control based on heterogeneous agent mirror learning, so that the traffic signal control task becomes a multi-agent sequential decision-making process, and design it as an online learning reinforcement learning method, combined with Transformer implementation. The self-attention mechanism of Transformer can well capture the high-level interaction relationship between multiple agents, and the autoregressive decoder naturally realizes the sequential decision-making process. The unified Transformer implementation can directly train the overall algorithm through a unified loss function. Use the vehicle queuing information for processing, simplifying the state and reward function design.
[0020] 2) The switching control strategy, sequential decision-making and simultaneous decision-making are combined into a reinforcement learning algorithm process, and the solvability, convergence and other properties of the algorithm are theoretically proved. Except for the agent of the global switching control strategy, the overall algorithm structure is implemented by Transformer, whether it is a sequential decision-making or a simultaneous decision-making scheme. A traffic signal control method with centralized training and decentralized execution of shared parameters is designed using Transformer. Then, a reinforcement learning global agent is additionally designed, and the switching control strategy is used for learning, so as to achieve the goal of unifying the three in a reinforcement learning algorithm process. The proposed multi-agent reinforcement learning selection method can prove that after adding the reinforcement learning global agent with an additional switching control strategy, the multi-agent system still converges to a solution of the original cooperative multi-agent reinforcement learning and has the property of strategy improvement. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A schematic diagram of a sequential decision-making scheme for multi-intersection traffic signal control according to Embodiment 1 of the present invention is shown;
[0022] Figure 2 A structural diagram of the MATLight algorithm model according to Embodiment 1 of the present invention is shown;
[0023] Figure 3 The road network structure diagram of the experimental data according to the embodiment 1 of the present invention is shown;
[0024] Figure 4 The module composition diagram of the LibSignal platform according to Embodiment 1 of the present invention is shown;
[0025] Figure 5 A comparison result diagram of Hangzhou according to Example 1 of the present invention is shown;
[0026] Figure 6 A Jinan comparison result diagram according to Example 1 of the present invention is shown;
[0027] Figure 7 A comparison result diagram of a single intersection according to Example 1 of the present invention is shown;
[0028] Figure 8 A diagram showing a trunk line flat peak comparison result according to Example 1 of the present invention is shown;
[0029] Fig. 9 A trunk peak comparison result diagram according to Example 1 of the present invention is shown;
[0030] Fig.10 A comparison result diagram of a single intersection according to Example 1 of the present invention is shown;
[0031] Fig.11A flow chart of a multi-agent network selection framework according to Embodiment 2 of the present invention is shown;
[0032] Fig.12 A flow chart of a multi-agent reinforcement learning selection algorithm according to Embodiment 2 of the present invention is shown;
[0033] Fig.13 A flow chart of a traffic signal control method for completely simultaneous decision making according to Embodiment 2 of the present invention is shown;
[0034] Fig.14 A comparison result diagram of Hangzhou according to Example 2 of the present invention is shown;
[0035] Fig.15 A Jinan comparison result diagram according to Example 2 of the present invention is shown;
[0036] Fig.16 A comparison result diagram of a single intersection according to Example 2 of the present invention is shown;
[0037] Fig.17 A diagram showing a trunk flat peak comparison result according to Example 2 of the present invention is shown;
[0038] Fig.18 A trunk peak comparison result diagram according to Example 2 of the present invention is shown;
[0039] Fig.19 A comparison result diagram of a single intersection according to Example 2 of the present invention is shown;
[0040] Fig. 20 The figure shows the ablation experiment result according to Example 2 of the present invention. DETAILED DESCRIPTION
[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0042] Embodiment 1:
[0043] An embodiment of the present invention provides a traffic signal control method, which obtains a heterogeneous agent mirror learning framework through multi-agent advantage decomposition theorem and heterogeneous agent mirror learning, and solves the constructed multi-intersection traffic signal control based on the heterogeneous agent mirror learning framework. During the solution process, a sequence model is introduced to obtain the final sequential decision-making plan.
[0044] Most of the existing excellent multi-agent reinforcement learning methods are based on the centralized training and decentralized execution paradigm. These methods cannot capture the full complexity of multi-agent interactions well, and it is difficult to alleviate the non-stationarity and convergence problems of multi-agent system strategy learning. In multi-agent reinforcement learning, even in a fully cooperative relationship, due to the existence of the credit allocation problem, it is impossible to clearly determine the contribution of each agent, so the strategy update directions of the agents may conflict with each other, causing the algorithm to fall into a suboptimal solution. Therefore, it is a challenge to ensure that each agent makes a separate decision while achieving a theoretically guaranteed strategy improvement on the joint strategy. The core theory of the theoretically guaranteed multi-agent trust region method is the multi-agent advantage decomposition theorem and the sequential strategy update scheme.
[0045] Theorem 1 (Multi-agent Dominance Decomposition Theorem): In any Markov cooperative game, let π be a joint strategy, (i1, ..., i m ) is an arbitrary sequence of multi-agents, for any state s and joint action The joint advantage function A of the agents π It can be decomposed using formula (1).
[0046]
[0047] This decomposition is natural and does not require any assumptions, such as MAPPO's assumption of shared agent parameters, QMIX's assumption of IGM conditions, etc. This theorem shows that the joint advantage value of multiple agents can be equal to the sum of the local advantage values of each agent after serialization.
[0048] The multi-agent dominance decomposition theorem points out a direction for finding performance improvements in multi-agent learning, namely, solving joint actions with positive dominance values. Specifically, the agents follow any order i 1:n Take actions in sequence. Assume that agent i1 takes action Make For the remaining agents m=2,...,n, agent i m Take Action Make Therefore, for a joint action a, the theorem guarantees that A π (s, a) is positive, thus ensuring that the performance is improved. In order to formally expand the above process into a policy iteration process with monotonic improvement guarantee, this study requires the following definitions and theorems.
[0049] Definition 1: Let π be a joint strategy, is the agent sequence i 1:m-1 A joint strategy is the intelligent agent i mTherefore, the objective function of reinforcement learning is expressed by formula (2).
[0050]
[0051] Based on Theorem 1 and Definition 1, the theorem of the trust region policy optimization algorithm can eventually be extended to MARL, thus obtaining the multi-agent trust region policy optimization method, namely Theorem 2.
[0052] Theorem 2: Let π be a joint strategy, then for any joint strategy get:
[0053]
[0054] Theorem 2 provides a perspective on how to improve the joint policy, that is, any positive update will lead to an increase in the sum. Since there are n agents making policy updates, the accumulated increments can be large, leading to substantial improvements. Note that this feature does not require a specific order for the agents to make updates, which allows for flexible scheduling of the update order at each iteration. This method is different from the trust region policy optimization method that directly applies a single agent to the joint policy of all agents. First, the multi-agent policy iteration method with monotonic improvement guarantee does not update the entire joint policy at once, but sequentially updates the individual policies of each agent. Second, during the sequential update process, each agent has a unique optimization objective that takes into account the updates of all previous agents, which is also the key to maintaining the monotonic improvement property. In summary, the multi-agent policy iteration method is obtained, as shown in Table 1.
[0055] Table 1 Multi-agent policy iteration with monotonic improvement guarantee via sequential updates
[0056]
[0057]
[0058] Multi-agent reinforcement learning methods usually do not have the monotonic improvement of rewards and unstable performance during convergence. In order to solve these problems, the sequential strategy update scheme based on the multi-agent advantage decomposition theorem can be classified into the new framework of heterogeneous agent mirror learning. Therefore, this framework can provide a general template for the design of multi-agent reinforcement learning algorithms. The algorithms derived from the heterogeneous agent mirror learning template all meet the properties of monotonic improvement of joint reward expectations and convergence to Nash equilibrium. To this end, this embodiment needs to first make the following definitions.
[0059] Definition 2 (drift functional): Let Drift functional of a heterogeneous agent It consists of a mapping defined by equation (4).
[0060]
[0061] Relative to π i The drift functional is defined as It said With π i The distance between them. Assuming that the agent sequence j 1:m The strategy has just been updated to Under this condition, the same update (from π i arrive ) can have different sizes, which will enable the algorithms to soft-constrain their learning steps in a collaborative way. Thus, this embodiment can define the advantage function under heterogeneous agent mirror learning.
[0062] Definition 3: Let is the drift functional of agent i, and the integrated advantage function of the heterogeneous agent mirror operator is defined by formula (5).
[0063]
[0064] when When , the heterogeneous agent mirror operator makes the evaluation value and advantage value of the heterogeneous agent drift functional different; when When , the value of the heterogeneous agent mirror operator is 0. Since the heterogeneous agent drift functional is non-negative, a strategy π to improve the heterogeneous agent mirror operator i It must be made positive, thereby improving the multi-agent advantage value of agent i. It is shown that the local improvement of an agent leads to the joint improvement of all agents. Therefore, Theorem 3 is obtained.
[0065] Theorem 3 (Heterogeneous Agent Mirror Operator): Let π old and π new For joint strategy, i 1:n ∈Sym(n) is the arrangement of multiple agents. For any state
[0066]
[0067] Therefore, for each state s, the union of π new Better than π old ,Right now We also naturally obtain the monotonic improvement property of the joint return J(π new )≥J(π old ).
[0068] Based on the above definitions and theorems, we can obtain the heterogeneous agent mirror learning algorithm framework as shown in Table 2.
[0069] Table 2 Heterogeneous agent mirror learning framework
[0070]
[0071]
[0072] This framework has excellent properties, including monotonically improving J(π k+1 )≥J(π k ), the value function converges to the Nash value function Expected return converges to Nash return The solution space consists of Nash equilibrium points and other properties, and the algorithms derived from this framework all satisfy the above properties.
[0073] In some cases, applying a single-agent reinforcement learning algorithm independently to each intersection may be a good choice. However, in most cases, this approach will have a negative impact on large intersection networks, so a collaborative learning method is needed to learn the best strategy for all road networks. Due to the connectivity of traffic signal components in multi-intersection road networks, multi-agent models provide more comprehensive and realistic solutions than single-agent models, and there is considerable potential to formulate traffic signal control problems as multi-agent systems. Therefore, multi-agent reinforcement learning methods are very suitable for the coordinated control of traffic signals at multiple intersections.
[0074] For the multi-intersection traffic signal control task, this embodiment regards it as a typical fully cooperative multi-agent system decision-making process. In this process, multiple intersections cooperate to complete traffic signal control decisions, thereby maximizing the overall goal, that is, maximizing the traffic efficiency of the entire road network and alleviating traffic congestion. For a road network with N intersections, the signal controllers of the N intersections in the environment can be described as agents. The goal of the reinforcement learning agent is to learn the best strategy for each agent and optimize the overall traffic flow conditions. In order to maximize the cumulative reward, each agent i must observe a part of the environment as its observation at each time step t. and take the best possible action based on the current traffic conditions observed at the intersection In the real world, Mostly predefined, i.e. traffic signals can only change between a few phases. The action will be performed in the environment, resulting in a reward The reward function can be defined at the level of a single intersection or a group of intersections in the environment.
[0075] Most of the existing methods focus on designing complex strategies or value network structures, state representations, and reward functions to improve the effect of multi-agent cooperative decision-making. However, these design methods often cannot guarantee the learning effect of multi-agents in theory, and can only obtain results through continuous debugging. Complex designs often bring difficulties in model training and have great limitations in application. In fact, in the problem of multi-intersection traffic signal control, the key to solving it is to clarify how to coordinate and control traffic signals among multiple intersections to achieve the goal of alleviating overall traffic congestion. In the actual process, the degree of coordination and influence between different intersections will be different. For example, usually, a certain intersection only has a greater impact on the intersections adjacent to it, while the intersections farther away have little impact. Therefore, in theory, there will be a clear ranking of the degree of influence among the intersections in the entire road network. Therefore, through a ranking like this, the interaction between multiple intersections in the control process can be captured. In addition to the interaction, in order to improve the final effect of traffic signal coordination control, it is necessary to further clarify the contribution of each decision of each intersection to the overall effect in the entire control process. This problem is called the credit allocation problem of multiple agents.
[0076] The reason why it is impossible to clearly define the credit allocation of multiple agents is that each agent cannot clearly know the contribution of its own strategy from the rewards, that is, it cannot know whether the increase in expected rewards is due to the fact that it has learned a better strategy or because other agents have learned a better strategy. Since multiple agents learn strategies at the same time, there are interactions between them, and the existence of the credit allocation problem leads to the instability of strategy learning, which may make the training of the entire system unable to converge. Therefore, after capturing the interactions between multiple agents, the key to this embodiment is to solve the credit allocation problem.
[0077] In essence, the credit allocation problem is the consequence of simultaneous decision-making by multiple agents. In the joint return of feedback, the algorithm cannot clearly identify the individual contribution of each agent, that is, the agent itself cannot know whether it has made a contribution. This problem is particularly prominent in the task of multi-intersection traffic signal control. The improvement in overall traffic efficiency cannot be clearly determined due to the improvement in traffic conditions at which intersection. Therefore, this embodiment proposes a solution to transform the control of multi-intersection traffic signals into a sequential decision-making process, that is, after intersection 1 makes a decision, intersection 2 makes a decision based on the decision of 1, and so on, so that the intersections in the entire road network complete the decision. Therefore, when intersection 1 makes a decision, the algorithm can exclude the interference of other agents, thereby obtaining the decision effect of intersection 1. Theoretically, in the coordinated control of traffic signals at multiple intersections, the action of the agent at the upstream intersection will affect the observation of the agent at the downstream intersection at the next moment. Therefore, the current action of the downstream intersection should take into account the impact it will receive. Therefore, this embodiment can regard the coordinated control of traffic signals at multiple intersections as a multi-agent sequential decision-making process. The multi-intersection traffic signal control process using a sequential decision-making scheme is as follows: Figure 1 shown.
[0078] Traffic signal control at multiple intersections can be modeled using a Markov cooperative game, which can be further defined as a decentralized partially observable Markov decision process.
[0079] · is a collection of multiple intersection agents.
[0080] Joint Observation Space is the product of the local observation space of the intersection. The state of the agent is defined at the level of a single intersection, which is equivalent to the definition of observation in MARL. Each agent observes a part of the environment. Exemplarily, the observation of the agent can be the queue length of the incoming lane. It corresponds to the number of vehicles waiting in this lane. Waiting vehicles are those with a speed less than 0.1m / s.
[0081] Joint Action Space is the product of the local action spaces of the agents at the intersection. At time step t, each agent chooses phase s as its action a t This means that the traffic signal controlled by the agent should be set to phase s. The agent selects one of eight non-conflicting candidate phases, and the best phase is selected by a reinforcement learning algorithm.
[0082] R: It is expressed as a joint reward function. The reward r of agent i iDefined as the negative value of the pressure at the intersection. The pressure at the intersection is defined as the absolute value of the sum of the pressures in the traffic flow directions. Logically, the greater the pressure, the greater the imbalance between the traffic flows in and out of the lanes. The pressure at intersection i is denoted as P i The traffic signal controller at intersection i is abstracted as agent i in MARL. So the reward is formulated as r i =-P i The agent aims to maximize the reward in order to stabilize the queue of vehicles in the system.
[0083] P: is the transition probability function.
[0084] γ∈[0,1) is the discount factor.
[0085] The above modeling is solved through a heterogeneous agent mirror learning framework with a sequential policy update scheme based on the multi-agent advantage decomposition theorem.
[0086] Heterogeneous agent mirror learning provides a design template for the algorithm of multi-agent sequential strategy update scheme. This framework can be used to derive many different MARL algorithms. This advantage allows this study to focus only on designing reasonable and feasible heterogeneous agent drift functionals, neighborhood operators, and sampling distributions. When the heterogeneous agent drift functional is set to When the neighborhood operator is set to the policy space of each agent and the sampling distribution is equal to the drift distribution, the Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) algorithm is derived, which is an extension of the PPO algorithm in MARL that is different from the MAPPO form.
[0087] The sequential strategy update method of the heterogeneous agent mirror learning framework brings a significant defect, that is, each agent updating the strategy in turn will increase the training time. Ideally, the designed algorithm hopes that each agent can train strategies in parallel, but due to the design of the sequential strategy update scheme, the derived HAPPO cannot be trained in parallel. Fortunately, the Transformer model combined with the sequential strategy update method of heterogeneous agent mirror learning perfectly solves this problem. The combined algorithm process perfectly matches the sequential multi-intersection traffic signal control process analyzed in this embodiment, and provides a solution algorithm for its sequential decision-making scheme. This embodiment calls the designed traffic signal control algorithm MATLight. This study uses the excellent sequence model of Transformer to perform unified training and learn the sequential decision-making strategy for multi-intersection traffic signal control.
[0088] The traffic signal control algorithm for sequential decision-making is composed of Transformer as the main architecture. The decoder of Transformer automatically generates actions for each agent in the order of the multi-agent sequence. The actions of each agent are generated in an autoregressive manner. This autoregressive solution process cleverly fits the sequential structure of the joint advantage function in Theorem 1, thereby well realizing the sequential update scheme. The encoder generates a representation sequence of joint observations, which will participate in the decoder's autoregressive calculation process, and the encoder will eventually obtain an estimate of the value function. The specific structure is as follows: Figure 2 shown.
[0089] Multi-agent input observation sequence and multi-agent output action sequences The mapping between can be considered as a sequence modeling task similar to machine translation. Therefore, the algorithm can use large sequence models like Transformer to perform collaborative multi-agent decision-making tasks. According to the multi-agent advantage decomposition theorem, the target action Depends on the decisions of all previous agents
[0090] The encoder parameters are denoted by φ, which transforms the observed sequence The observations are fed to the encoding block. Each encoding block contains a self-attention mechanism, a multi-layer perceptron (MLP), and residual connections to prevent gradient vanishing and network degradation as the network depth increases. The output of the observations is encoded as It not only affects the agent (i1, ..., i n ) information, including the high-level interaction relationship that reflects the interaction of intelligent agents. During the training phase, the encoder uses equation (7) to approximate the value function that minimizes the Bellman error.
[0091]
[0092] Where T represents the total number of time steps and n represents the number of agents. At each time step t, the joint reward R(o t , a t ) is from the agent sequence in the observation sequence o t Execute action sequence a t The loss function L Encoder (φ) represents the predicted V value of the encoder network forward propagation The time difference project target V value calculated with the Bellman equation The mean square error between .
[0093] The parameters of the decoder are denoted by θ. is transmitted from the decoder to the sequence of decoding blocks. Each decoding block has a masked self-attention mechanism that is only used for the i-th r and the i j The attention is calculated between the action heads, r < j to ensure that the sequential update strategy is maintained throughout the calculation process. Then a second masked attention mechanism is used to calculate the attention between the action head and the observation representation. Finally, the decoding block ends with an MLP and residual connection. The output of the last decoding block is a joint action sequence representation It is input to an MLP whose output target is a probability distribution, i.e., the policy To train the decoder, the algorithm minimizes the following objective of the proximal policy optimization clipping algorithm (PPO-clip) using equation (8).
[0094]
[0095] This example uses the generalized advantage estimate as a robust estimate of the joint advantage function The calculation formula of generalized advantage estimate is: Here γ and λ are two hyperparameters in the generalized advantage estimation, which are used to adjust the influence of variance and bias. is the time difference error term. Clip is the truncation operation, which means Restricted to the interval [1-∈, 1+∈], ∈ indicates the range of execution truncation, Represents the difference between the old and new strategies of the agent. MATLight follows the basic framework of heterogeneous agent mirror learning, so it still has its good properties. The process of the traffic signal control method combined with Transformer multi-agent sequential decision-making proposed in this embodiment is shown in Table 3.
[0096] Table 3 Traffic signal control combined with Transformer multi-agent sequential decision making
[0097]
[0098] The feasibility and advancement of the present invention will be demonstrated by combining experimental results and analysis.
[0099] Experimental platform:
[0100] (1) Traffic Micro-Simulator
[0101] This study uses a simulation platform to simulate the operation of traffic flow in the urban road network to carry out experiments. This embodiment deploys the road network and traffic flow based on the CityFlow traffic micro-simulator, and uniformly conducts training and testing on the LibSignal platform, which includes the CityFlow micro-traffic simulator and also provides a unified testing framework, interface and evaluation indicators. After entering the traffic data in the simulator, the vehicle drives to the destination according to the environment settings. The decision of the signal control method is executed by the simulator, which also provides traffic status and evaluation methods. By convention, after each green light phase, there is a three-second yellow light phase and a two-second full red phase.
[0102] (2) Traffic Dataset
[0103] In order to test the universality of the algorithm to a certain extent, the algorithm experiments are divided into two categories: network-level control and trunk line control. In the network-level experiment, this study used real data sets of the road networks in Hangzhou and Jinan. Since the two data sets are different in road network structure and traffic flow characteristics, the universality of the algorithm of this embodiment can be verified to a certain extent. In the trunk line control experiment, this study used synthetic traffic flow data of 6×1 roads. In order to verify the adaptability of the algorithm to different traffic flows, this study selected two types of data: flat peak and peak.
[0104] Hangzhou 4×4: There are 16 intersections on Gudang Street in Hangzhou, China. Roadside surveillance cameras collect important records of traffic data. The trajectories of vehicles through these operations are recorded on a computer by using the camera position analysis records. Therefore, in this study, the number of vehicles passing through these intersections is used as the traffic volume. Due to the low quality of real-world data, the data is necessarily simplified. The traffic volume is from the camera data in Hangzhou, and the turn ratio is set to 10% (left turn), 60% (straight ahead), and 30% (right turn), which is obtained based on the statistical synthesis of taxi GPS data.
[0105] Jinan 3×4: Similar to Hangzhou 4×4, traffic data is collected and processed by roadside cameras near 12 intersections on Dongfeng Street in Jinan, China, in the same way as Hangzhou 4×4.
[0106] Mainline 6×1: The synthetic dataset contains both off-peak and peak traffic flows. This information is derived from a statistical analysis of actual traffic behavior in Jinan and Hangzhou. All vehicles enter and leave the network from the edge of the road network. For each entering edge, for off-peak traffic flow, the number of vehicles generated is sampled from a Gaussian distribution with a mean of 300 vehicles / hour / lane and a standard deviation of 0.3, while for peak traffic flow, the number of vehicles generated is sampled from a Gaussian distribution with a mean of 300 vehicles / hour / lane and a standard deviation of 0.6. The standard deviation represents the degree of dispersion of traffic data, that is, the degree to which the traffic value deviates from the mean. The difference in standard deviation can reflect the volatility of traffic, and thus indirectly reflects the off-peak and peak traffic conditions. That is, a larger standard deviation indicates a larger traffic fluctuation, showing a peak traffic situation; while a smaller standard deviation indicates a smaller traffic fluctuation, representing an off-peak traffic situation. The turning ratios are all set to 10% (left turn), 60% (straight ahead), and 30% (right turn).
[0107] For the road network structures of the three traffic datasets, Figure 3 shown.
[0108] (3)LibSignal library
[0109] LibSignal is a highly unified, extensible, and comprehensive library for traffic light control tasks. The library collects and filters 9 commonly used datasets and implements 10 different baseline models in two influential traffic simulators, including SUMO and CityFlow. The library provides PyTorch implementations of some paper algorithms, and can achieve the same performance level as the original TensorFlow official code, and calibrates the simulator to improve reliability across simulator environments. In addition, LibSignal allows for a unified and fair comparison of the performance of different algorithms under various datasets and simulators. LibSignal provides benchmark results and discussions for traffic signal control for researchers interested in this topic. The structure of the platform is as follows Figure 4 shown.
[0110] Evaluation Metrics
[0111] This study uses Average Traveling Time (ATT) and Average Intersection Pressure (AIP) to evaluate the performance of various TSC models. The total time (in seconds) required for all vehicles to arrive and leave the area is averaged to calculate the average travel time. It is the most common performance metric for TSC problems in the transportation field. Intersection pressure indicates the degree of imbalance in vehicle density between inbound and outbound lanes. Minimizing pressure can be effectively used to optimize throughput.
[0112] In order to prove that MATLight has the ability of coordinated control, this study selects the intersection with the largest traffic volume in the road network and observes the impact of MATLight on the traffic conditions of a specific intersection. The impact of the algorithm on the traffic conditions of a specific intersection is measured by looking at the average queue length of a single intersection. The results of intersection selection are shown in Table 4. For example, ID 1_4 is defined as the intersection represented by the first column from west to east and the fourth row from south to north in the road network.
[0113] Table 4 Intersections with the highest traffic volume (vehicles / hour)
[0114]
[0115] Experimental setup
[0116] This study sets the memory capacity of the replay experience pool of PressLight, CoLight, and MPLight to 360. This is the same as the number of decisions made by the agent in a one-hour simulation. MATLight has only one complete 60-minute trajectory for each policy update. This allows the reinforcement learning model to learn in a small sample space that is basically consistent. In this case, MATLight has a higher sampling efficiency than CoLight, MPLight, and PressLight.
[0117] To test the stability, each reinforcement learning algorithm was repeated 5 times on each data set, and the results were averaged. The shaded area between the curves is the standard error interval, which indicates the stability of the algorithm. For traditional methods such as FixedTime, MaxPressure, and SOTL, there are no relevant curves in the experiment because there is no learning process. Because traditional algorithms do not have active coordinated control, there is no need to observe their impact on specific intersections.
[0118] Performance and analysis
[0119] The results of ATT and AIP comparing the traditional method and the reinforcement learning method at the network level are shown in Table 5. According to the experimental results, the algorithm constructed in this embodiment outperforms the compared baseline methods in both ATT and AIP indicators of the two traffic datasets.
[0120] Table 5 Evaluation of different methods for network-level control
[0121]
[0122] As the number of rounds increases, the ATT curves of MPLight, CoLight, and MATLight in network-level control are as follows: Figure 5 (a) and Figure 6As shown in (a) in the Hangzhou dataset, MATLight's strategy is slightly unstable, but the final convergence result and convergence speed are still better than MPLight and CoLight. On the Jinan dataset, MATLight is completely better than MPLight and CoLight. Their AIP curves are shown in Figure 5 (b) and Figure 6 As shown in (b). As the average intersection pressure value decreases, the distribution of vehicles in the entire system becomes more uniform. The trends of the AIP curves of MPLight, CoLight, and MATLight are roughly consistent with their ATT curves. This shows that the design using intersection pressure as a reward function can well optimize the final goal of TSC.
[0123] The average queue lengths of MPLight, CoLight, and MATLight algorithms at a single intersection on a network-level road tend to converge as the number of rounds increases, and MATLight has the best convergence result. Figure 7 As shown in (a), MPLight has the slowest convergence speed at Hangzhou 1_4 intersection, while MATLight has the fastest. Figure 7 As shown in (b), CoLight shows obvious instability at Jinan 1_1 intersection, while MATLight has better convergence than MPLight. The average travel time curve and the single intersection queue length curve are combined for analysis. The results show that the strategy learned by the MATLight algorithm designed in this embodiment in the multi-intersection traffic signal control task has the intention of coordinated control.
[0124] Table 6 shows the ATT and AIP results of different algorithms on trunk control. According to the experimental results, the algorithm proposed in this embodiment performs better than the baseline method for comparison in both ATT and AIP on both flat and peak traffic data sets. Traditional methods include FixedTime, MaxPressure and SOTL, and PressLight is a baseline method for reinforcement learning.
[0125] Table 6 Evaluation of different methods of trunk control
[0126]
[0127]
[0128] Figure 8 (a) and Fig. 9(a) shows the ATT curves of PressLight and MATLight in trunk line control as the number of rounds increases. Although MATLight has certain instability in trunk line control, its convergence speed and average travel time values are better than PressLight under both flat and peak traffic conditions. A more reasonable explanation for this is that the algorithm proposed in this embodiment rearranges the order of the multi-agents in the sequence each time it is trained to avoid overfitting of the algorithm and thus obtain a Nash equilibrium. After the strategy converges, a new multi-agent sequence may cause the algorithm to explore a new joint strategy, resulting in a sudden drop in performance, but after a period of training, the algorithm's performance can quickly return to its original level. The algorithm proposed in this embodiment is superior to PressLight not only in AIP on flat traffic flow, but also in peak traffic flow. The trends of the AIP curves of PressLight and MATLight are roughly equivalent to their ATT on flat and peak traffic, as shown in Figure 2. Figure 8 (b) and Fig. 9 As shown in (b). Fig.10 Indicates the control effect of the algorithm on a specific intersection in the arterial coordination. In the arterial coordination of the 6×1 road network, MATLight has a better control effect than PressLight at a single intersection in both off-peak and peak traffic flows. Fig.10 In (a), at the 2_1 intersection on the flat peak, the queue length value of MATLight is lower than that of PressLight, but there are some fluctuations. Fig.10 As shown in (b), the queue length of MATLight at the peak intersection 5_1 is not only lower than that of PressLight, but also quite stable. When the ATT curve is combined with other curves for analysis, even if MATLight shows a slight jitter in performance in trunk control, under the coordinated control of the algorithm design, the intelligent agent will not cause traffic fluctuations at the intersection with the highest traffic flow demand.
[0129] Therefore, this embodiment first introduces the relevant theories and properties of the heterogeneous agent mirror learning framework based on the multi-agent advantage decomposition theorem and the sequential strategy update scheme, and then proposes how to model the multi-intersection traffic signal control problem by sequential decision-making, thereby transforming it into a multi-agent sequential decision-making process, and gives the process by which the control problem can be designed based on heterogeneous agent mirror learning. Then, a traffic signal control algorithm based on the combination of heterogeneous agent mirror learning and Transformer is designed, and it is proved that this algorithm design by optimizing credit allocation has excellent properties such as joint strategy monotonic improvement and convergence, and experiments are carried out on different public traffic data sets. The experimental results show that the proposed method has better control effect, faster convergence speed and better small sample learning efficiency.
[0130] Embodiment 2:
[0131] The embodiment of the present invention provides a traffic signal control method. On the basis of the method provided in Example 1, the credit allocation problem, strategy instability and convergence problem of coordinated control of multiple signal lights are studied to solve the problem that in the coordinated control task of traffic signals at multiple intersections, different convergence results may be brought about due to the addition of constraints on the update direction of multi-agent strategies, so it may be difficult to learn and obtain the optimal solution of the system, that is, the Nash equilibrium point of the random game. Moreover, for a multi-agent system, there are multiple Nash equilibrium points, and the algorithm can easily fall into the local optimum. Therefore, how to avoid the negative impact brought by adding constraints is a key. The method proposed in the previous chapter optimizes the credit allocation through the sequential decision modeling control process, alleviates the non-stationarity and convergence problems of the joint strategy learning of coordinated control of traffic signals at multiple intersections, but adds constraints in the modeling, so there is still room for improvement in the solution space. In this embodiment, in order to prevent the joint strategy learned by the algorithm from falling into the local optimal solution, this embodiment proposes a selective multi-agent strategy theory based on the multi-agent network selection framework, and designs a multi-agent reinforcement learning selection method to further fully explore and utilize the advantages of multi-agent sequential decision-making schemes and simultaneous decision-making schemes, thereby improving the effect of traffic signal control.
[0132] In the multi-agent decision-making process under full cooperation, in some cases, it is sufficient to choose the simultaneous decision-making scheme without considering the impact of other agents' decisions. In some cases, it is necessary to consider the decisions of other agents so that one can make better decisions. At this time, the multi-agent sequential decision-making scheme is better. Therefore, in order to enable the multi-agent system to automatically learn when to make simultaneous decisions and when to make sequential decisions, an additional agent based on reinforcement learning must be designed to learn how to achieve this goal. This embodiment proposes a selective multi-agent strategy theory and designs a multi-agent reinforcement learning selection algorithm, creatively combining the two schemes of multi-agent sequential decision-making and simultaneous decision-making, and proves that the overall system is still a problem that can be solved by reinforcement learning, and the result converges to a solution of the original system. After multiple verifications on urban traffic data sets and micro traffic simulator simulations, it is shown that this algorithm has achieved better results in the problem of coordinated control of multiple signal lights.
[0133] The multi-agent network selection framework is designed to learn to determine when to use the centralized learning process to improve the learning and performance of multi-agents. The framework contains N multi-agent reinforcement learning agents, a centralized learner, an independent learner, and an adaptive reinforcement learning agent. Among them, the adaptive reinforcement learning agent decides in which states of the training phase to activate the centralized learner and use the independent learner in other states.
[0134] Each multi-agent reinforcement learning agent has two value-based policies, one policy that obtains joint actions by inputting the global state of global observations, and the other policy that only inputs local observations. Independent Q-Learning (IQL) is an independent learner that trains each agent's independent policy; while QMIX is a centralized learner that trains a centralized critic network. Adaptive reinforcement learning agents use an extended form of Q-learning, and their policy action set contains two actions, using centralized training or not. Adaptive reinforcement learning agents update their own policies as each agent learns the policies of individuals. And the framework only allows centralized updates when the adaptive reinforcement learning agent performs a centralized learner call, which ensures that the communication cost between agents during training is strictly limited to when the adaptive reinforcement learning agent performs a centralized learner activation. Fig.11 is a flowchart of the framework.
[0135] In fact, during the entire execution process of a task, it is not always necessary for multiple agents to follow the sequential decision-making scheme. In order to fully utilize the respective advantages of both multi-agent simultaneous decision-making and sequential decision-making schemes, this embodiment can additionally design an agent with global switching control. The global agent determines when multiple agents make decisions sequentially or simultaneously, and through the scheduling of the global agent, the learning and performance of the multi-agent system are improved. To this end, this embodiment proposes a selective multi-agent strategy theory and designs a new multi-agent reinforcement learning selection algorithm. The algorithm includes an additional global reinforcement learning agent as a switching controller. This global agent uses a switching control strategy and selectively applies simultaneous multi-agent reinforcement learning only when sequential multi-agent reinforcement learning may fall into a local optimum. In this algorithm, the simultaneous decision-making and sequential decision-making components are independent of each other, the playback experience pool is shared, and the shared experience is used for the training of the three components of global, sequential and simultaneous.
[0136] The multi-agent reinforcement learning selection algorithm (MARL-NS) of this embodiment introduces a new global adaptive reinforcement learning agent that learns when n MARL agents are trained, and it uses a switching control strategy that is different from the standard reinforcement learning framework. Since the additional global agent has different goals from the n agents, this may cause convergence problems and cause the original cooperative Markov game to turn into a non-cooperative Markov game. To model. The specific modeling is as follows:
[0137] · is the collection of MARL agents. G is the global agent.
[0138] The joint observation space is the product of the local observation spaces of the agents, expressed as
[0139] The joint action space is the product of the local action spaces of multiple agents, expressed as is the set of actions of the global agent.
[0140] R: is the joint reward function. R G =Rc·1(g) is the reward function of the global agent.
[0141] P: is the transition probability function.
[0142] γ∈[0,1) is the discount factor.
[0143] In the process of game modeling, the main components of the multi-agent reinforcement learning selection algorithm are the sequential decision MARL algorithm, the global switching control reinforcement learning algorithm, and the simultaneous decision MARL algorithm. The specific algorithm structure is as follows: Fig.12 Although it may cause convergence problems in theory, this embodiment proves the convergence of the algorithm in the next section and obtains relevant theories. The algorithm can still be solved by reinforcement learning, and through the designed new Bellman operator, the algorithm converges to a solution of the original cooperative multi-agent system problem.
[0144] In order to complete the traffic signal control of learning two strategies of simultaneous decision-making and sequential decision-making, this embodiment needs to design a simultaneous decision-making algorithm equivalent to the MATLight sequential decision-making algorithm. This embodiment is based on the centralized training and decentralized execution paradigm and follows the relevant settings of MAPPO to design a simultaneous decision-making algorithm. The algorithm structure is as follows Fig.13 shown.
[0145] The critic module and actor module of the algorithm are both composed of the encoder part of the Transformer. During the training process, all agents share a centralized critic network to evaluate the current value of each agent. During execution, each agent selects actions based only on observations, and adopts the MAPPO shared parameter design, where the agents share the actor network parameters.
[0146] The main components of the traffic signal control algorithm combined with multi-agent reinforcement learning selection are composed of a sequential decision-making MARL algorithm, a global switching control reinforcement learning algorithm, and a simultaneous decision-making MARL algorithm.
[0147] Fully Sequential Decision Making: This component combines the algorithms of the Transformer and heterogeneous agent mirror learning frameworks to implement a same-strategy online learning method for sequential multi-intersection TSC.
[0148] Fully simultaneous decision-making: For simultaneous decision-making, this embodiment provides a centralized training decentralized execution mode and replaces the autoregressive decoder part with an encoder. The encoder is shared among multiple agents, and its input is the local observation of the agent.
[0149] Global switching control agent: The action set of the global strategy consists of two actions: 1) using a sequential decision strategy; 2) using a simultaneous decision strategy. The global state is input to determine whether to choose a sequential decision strategy or a simultaneous decision strategy.
[0150] In MARL-NS, the global agent observes the global state s of the environment t , and samples the discrete strategy of the switching controller First, the global agent estimates the state s t The global agent then uses its own strategy to Get g t If g t = 1, the algorithm chooses sequential decision making to obtain joint actions. t = 0, then a joint action a is generated from the simultaneous decision scheme t . It is particularly important to note that the sequential strategy and the simultaneous strategy are independent of each other. The experience generated by the above process is stored in the replay experience pool and is used to train the three modules of the global agent, sequential and simultaneous decision-making. The global agent achieves the goal of improving the learning process and maximizing the performance of the N MARL agent system by deciding where to use sequential decisions. In order to encourage the global agent to selectively choose when to make sequential decisions, each use of the sequential decision-making scheme will bring a fixed intervention cost to the global agent, which is measured by a constant c>0.
[0151] In order to enable the new system to be trained through reinforcement learning methods, this embodiment needs to define a learning method for an additional global agent.
[0152] Definition 4 (Strategy of the global agent): For any state and all joint actions Global Agent Strategy It is given by formula (9).
[0153]
[0154] Equation (4-1) characterizes where the global agent should execute sequential decision-making schemes. During the learning phase, this condition allows the algorithm to evaluate the representation online. This measure is the expected return of the global agent minus its intervention cost after switching to the joint strategy of sequential decision-making, and the final decision result is obtained through the indicator function.
[0155] Definition 5 (Extended Q-learning variant): This embodiment uses formula (10) to expand the update rule of the Q-learning variant.
[0156]
[0157] This embodiment uses a neural network to approximate the result.
[0158] Therefore, this embodiment uses the extended deep Q network of Definition 5 to train the global agent of MARL-NS. Specifically, the global agent minimizes the loss function defined by equation (11).
[0159]
[0160] The summary of the MARL-NS algorithm is shown in Table 7.
[0161] Table 7 Learning simultaneous and sequential multi-intersection traffic signal control
[0162]
[0163] For a set of joint sequential and simultaneous decision strategies, the solution to the new problem modeled in this chapter is a set of limit points compressed by the Bellman operator acting on the value function in Theorem 4.
[0164] Theorem 4: Let v G : For any joint strategy π s , the solution to the new modeling problem is provided by equation (12).
[0165]
[0166] In formula (12), the Bellman operator is Given a convergent MARL learning system, when the global agent uses a Q-learning variant, it ensures the convergence of the system. Then the algorithm can achieve the performance results sought by the design. There exists some finite integer N, For any m≤N, such that and π m is the joint strategy after the mth learning iteration, with and without the influence of the global agent. The proof is in Appendix A.
[0167] Theorem 4 shows that the results of a multi-agent system with sequential decision making, simultaneous decision making, and global switching control trained simultaneously converge to a solution to the original MARL problem. The MARL-NS framework can be used to improve the underlying MARL system.
[0168] The feasibility and advancement of this embodiment will be further illustrated below in conjunction with specific experiments.
[0169] The experimental environment and settings are the same as those in Example 1. The experiment is still divided into two groups: network-level control and trunk control. The difference is that in order to test the control effects of MPLight, CoLight and PressLight in their non-targeted design task scenarios, this example tests MPLight and CoLight on the trunk control dataset. Similarly, PressLight is also tested on network-level control. All algorithms still learn in the same small sample space size. For the global switching control agent of MARL-NS, the playback experience pool of the same sample size is also used for training. All algorithms are repeated 5 times on all datasets to verify the stability of the algorithm.
[0170] Analysis of evaluation index results on the dataset
[0171] In Table 8, this embodiment compares the control effects of MARL-NS proposed in this embodiment with traditional traffic signal control methods (FixedTime, MaxPressure, and SOTL) and reinforcement learning-based baselines (CoLight, MPLight, and PressLight) in the traffic flow dataset of the network-level control task. The results show that MARL-NS consistently outperforms all other methods in terms of ATT and AIP for all datasets. Moreover, for the queue length value at a single intersection, MARL-NS also shows results that are superior to other reinforcement learning baseline methods.
[0172] Table 8 Evaluation of different methods of network-level control
[0173]
[0174] Fig.14 and Fig.15 They are the comparative experimental results on the Hangzhou and Jinan datasets. Fig.14 and Fig.15 In (a), the ATT curve is plotted as the number of rounds increases. On the Hangzhou and Jinan datasets, MARL-NS shows superior convergence speed and stability compared to other algorithms. Fig.14 and Fig.15 The corresponding AIP curve in (b) shows that the MARL-NS algorithm has a more effective vehicle distribution, and the trend of the AIP curve of MARL-NS is consistent with its ATT performance. Fig.16 The comparison results of queue length at a single intersection of different algorithms in network-level control are shown. Fig.16 (a) shows that MARL-NS achieves the fastest convergence speed at the intersection 1_4 in Hangzhou. At the intersection 1_1 in Jinan, Fig.16 As shown in (b), MARL-NS shows superior results in terms of convergence speed and stability compared to the obviously unstable CoLight, slower MPLight and PressLight.
[0175] Table 9 shows the evaluation results of the algorithm proposed in this embodiment and the comparison algorithm on the trunk line control task dataset. The results show that MARL-NS is always better than other baseline algorithms in both flat-peak traffic and peak traffic conditions.
[0176] Table 9 Evaluation of different methods of trunk control
[0177]
[0178] Fig.17 and Fig.18 These are the experimental results for the trunk flat peak and trunk peak comparison. Fig.17 and Fig.18 (a) shows that MARL-NS significantly exhibits faster and more stable convergence in terms of ATT under both flat and peak traffic conditions. The superior performance trend of MARL-NS is further confirmed by the AIP curves under flat and peak traffic conditions, such as Fig.17 and Fig.18 In addition, Fig.19 is the comparison result of a single intersection. Fig.19 (a) and (b) show that MARL-NS maintains a lower and more stable queue length at intersections 2_1 and 5_1 for both flat and peak traffic conditions. A comprehensive analysis of the queue length curve and the average travel time curve at a single intersection shows that the algorithm proposed in this embodiment performs effective coordinated control.
[0179] The method proposed in this embodiment is an important part of the real-time deployment system of intelligent traffic signal control. The response time directly affects its practical application value. The response time refers to the time interval from the system receiving input data to generating control decisions and executing them. Specifically, the response time can be divided into four stages: data acquisition time, data processing time, decision time and execution time. Among them, the data acquisition time is the time to collect data from the sensor, the data processing time is the time to preprocess and analyze the collected data, the decision time is the time for the algorithm to generate control decisions, and the execution time is the time to pass the decision to the traffic light and execute it. Therefore, the complete response time is expressed as the sum of the four stages. Since this study is based on the algorithm development and testing of the microscopic traffic simulator, the data acquisition time and execution time are determined by the equipment and network of the real-world traffic system, which are ignored in this embodiment. The data processing time and decision time are determined by the algorithm proposed in this embodiment, and the two are unified and integrated into a whole by the model. Therefore, the algorithm response time verification of this embodiment only considers the sum of the data processing time and decision time, that is, the time from the algorithm receiving input data to the final output result. The evaluation results are shown in Table 10. The response time is the average response time of the algorithm model in one round after the control strategy training is stable. The experimental results show that the response time of the MARL-NS method proposed in this embodiment is less than 0.05 seconds on all experimental data sets, which meets the deployment requirements of real-time traffic signal control systems.
[0180] Table 10 Response time evaluation
[0181]
[0182] Ablation experiment
[0183] In order to explore the effectiveness and necessity of the global switching control strategy, this embodiment conducts relevant ablation experimental research. To this end, this embodiment implements two algorithms, namely, fully sequential decision-making and fully simultaneous decision-making. The results of the ablation experiment are shown in Table 11. For the ATT results, MARL-NS is better than the fully sequential and fully simultaneous decision-making schemes.
[0184] Table 11 Evaluation of ablation experiments
[0185]
[0186] In terms of convergence and stability, MARL-NS is also better than the fully sequential and fully simultaneous decision-making schemes. The results of the average travel time ablation experiment are as follows: Fig. 20 This shows that after adding global switching control, the overall control effect of the system has been improved.
[0187] In summary, this embodiment first explains the defects of the traffic signal control method of the fully sequential decision-making scheme proposed in Example 1, and then proposes a selectable multi-agent strategy theory based on the multi-agent network selection framework, and designs a multi-agent reinforcement learning selection algorithm, thereby combining the sequential decision-making scheme with the simultaneous decision-making scheme, and proves that the system of reinforcement learning agents with the additional global switching control strategy still converges to a solution to the original MARL problem, and the solution process can still be solved by reinforcement learning. Based on this, this chapter further proposes a multi-intersection traffic signal control method that learns to select sequential decisions and simultaneous decisions, and trains and tests it on different relevant public traffic data sets. The experimental results show that the proposed method further improves the traffic signal control effect, and the learned strategy has better stability.
[0188] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.
Claims
1. A traffic signal control method, characterized in that: The method comprises: Determine the joint strategy based on the multi-agent advantage decomposition theorem; Determining a heterogeneous agent mirror learning framework based on the joint strategy; Construct a multi-intersection traffic signal control model, wherein the multi-intersection traffic signal control model includes a set of multi-intersection agents Joint Observation Space Joint Action Space Joint Reward Function Transition probability function and a discount factor γ∈[0,1), the joint observation space is the product of the local observation spaces of the intersection, the joint action space is the product of the local action spaces of the agents at the intersection, and the reward r of the i-th agent is i is the negative value of the pressure at the intersection. The pressure at the intersection is defined as the absolute value of the sum of the pressures in the traffic flow direction. The pressure at the i-th intersection is expressed as P i , the traffic signal controller at the i-th intersection is abstracted as the i-th agent in the set of multi-intersection agents, and the reward formula is expressed as r i =-P i ; The multi-intersection traffic signal control model is solved using the heterogeneous agent mirror learning framework to obtain a final strategy.
2. The method according to claim 1, characterized in that Determine the joint strategy based on the multi-agent advantage decomposition theorem, including: For any state s and joint action The joint advantage function A of the agents π Decomposed by the following formula: In the formula, Represents the agent sequence i 1:m The joint advantage value of represents the individual advantage value of the jth agent in the agent sequence, Represents the agent sequence i 1:j-1 The joint action represents the action of the jth agent in the agent sequence, j represents the jth agent, and m represents a total of m agents; Let π be a joint strategy, is the agent sequence i 1:m-1 A joint strategy is the intelligent agent i m A strategy for reinforcement learning is expressed by the following formula: In the formula, represents the objective function of reinforcement learning, ρ π represents the state distribution, For any joint strategy satisfy: In the formula, represents the objective function value under the new reinforcement learning strategy, j(π) represents the objective function value under the old strategy, n represents n agents, Represents the KL divergence value of the new and old strategies multiplied by the coefficient C.
3. The method according to claim 1, characterized in that In the heterogeneous agent mirror learning framework, the drift functional of a heterogeneous agent is consists of a mapping represented by Relative to π i The drift functional is defined as express With π i The distance between agents in the sequence j 1:m The strategy has just been updated to In the case of i arrive Available in different sizes, represents the state space, represents the transition probability without agent i, represents the drift functional between the new and old strategies, represents the probability distribution of advantage values; The advantage function of the heterogeneous agent mirror operator integration is expressed as: In the formula, represents the heterogeneous agent mirror operator, A π represents the joint advantage value, Express expectations, represents the state distribution, β π (s) represents the sampling distribution; when When , the heterogeneous agent mirror operator makes the evaluation value and advantage value of the heterogeneous agent drift functional different; when When , the value of the heterogeneous agent mirror operator is 0; Let π old and π new For joint strategy, i 1:n ∈Sym(n) is the arrangement of multiple agents. For any state satisfy: In the formula, represents the mirror operator of heterogeneous agents under the new strategy, represents a new joint strategy for multiple agents, represents the joint advantage value under the old strategy, represents the heterogeneous agent mirror operator under the old strategy, Indicates the new federation strategy.
4. The method according to claim 1, characterized in that: Constructing a sequence model, when solving the multi-intersection traffic signal control model using the heterogeneous agent mirror learning framework, the multi-agent input observation sequence and multi-agent output action sequences The mapping between them is taken as a sequence modeling task, and the sequence model is used to perform the sequence modeling task.
5. The method according to claim 4, characterized in that The sequence model includes an encoder and a decoder; The encoder includes at least one encoding block, each encoding block includes a self-attention mechanism, a multi-layer perceptron and a residual connection, and the encoding block uses a multi-agent input observation sequence As input, the output of the observations is encoded as The encoder approximates the value function that minimizes the Bellman error by the following formula: In the formula, T represents the total number of time steps, n represents the number of agents, and at each time step t, the joint reward R(o t ,a t ) is from the agent sequence in the observation sequence o t Execute action sequence a t Feedback from the back environment, loss function L Encoder (φ) represents the predicted V value of the encoder network forward propagation The time difference project target V value calculated with the Bellman equation The mean square error between them, φ is the parameter of the encoder; Embedding of joint actions is transmitted from the decoder to a sequence of decoding blocks, each of which includes a masked self-attention mechanism that is only used in the i-th r and the i j The attention is calculated between the action heads, r < j ensures that the sequential update strategy is maintained throughout the calculation process, and the attention between the action head and the observation representation is calculated using the second masked attention mechanism. The decoding block ends with an MLP and residual connection. The output of the last decoding block is a joint action sequence representation. It is input into a multilayer perceptron whose output target is a probability distribution, and the strategy is obtained The decoder is trained using the following formula: in In the formula, the generalized advantage estimate is used as the robust estimate of the joint advantage function The calculation formula of generalized advantage estimate is: γ and λ are two hyperparameters in the generalized advantage estimation, which are used to adjust the influence of variance and bias. is the time difference error term, and clip is the truncation operation, which means Restricted to the interval [1-∈, 1+∈], ∈ indicates the range of execution truncation, represents the difference between the agent’s old and new strategies, θ is the parameter of the decoder, is the intelligent agent i m New strategies, is the intelligent agent i m The old strategy, is the intelligent agent i m Actions, is the agent sequence i 1:n The joint observation representation of is the agent sequence i 1:m-1 Joint action representation.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Introducing a global agent, the global agent is used to determine when multiple agents make decisions sequentially or simultaneously, and after introducing the global agent, the multi-intersection traffic signal control model also includes a reward function of the global agent; The global agent estimates the state s t The Q value of the sequential and simultaneous decision-making schemes is obtained by the global agent through its own strategy. t , if g t = 1, then select sequential decision making to obtain joint actions. If g t = 0, then a joint action a is generated from the simultaneous decision scheme t .
7. The method according to claim 6, characterized in that For any state and all joint actions Global Agent Strategy Given by the following formula: In the formula, It is a measure of the expected return of the global agent minus its intervention cost after switching to the joint strategy of sequential decision-making, and the final decision result is obtained through the indicator function. represents the interference factor, Q G (s t , a t ) represents the joint Q value, Q G (s t , a t |π, g) represents the joint Q value of multiple agents without intervention in the global agent decision, and c represents the intervention cost.
8. A traffic signal control device, characterized in that: The apparatus comprises a processor configured to: Determine the joint strategy based on the multi-agent advantage decomposition theorem; Determining a heterogeneous agent mirror learning framework based on the joint strategy; Construct a multi-intersection traffic signal control model, wherein the multi-intersection traffic signal control model includes a set of multi-intersection agents Joint Observation Space Joint Action Space Joint Reward Function Transition probability function and a discount factor γ∈[0,1), the joint observation space is the product of the local observation spaces of the intersection, the joint action space is the product of the local action spaces of the agents at the intersection, and the reward r of the i-th agent is i is the negative value of the pressure at the intersection. The pressure at the intersection is defined as the absolute value of the sum of the pressures in the traffic flow direction. The pressure at the i-th intersection is expressed as P i , the traffic signal controller at the i-th intersection is abstracted as the i-th agent in the set of multi-intersection agents, and the reward formula is expressed as r i =-P i ; The multi-intersection traffic signal control model is solved using the heterogeneous agent mirror learning framework to obtain a final strategy.
9. A traffic signal control system, characterized in that: The system comprises: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7. 10 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform the method according to claim 1 .
Citation Information
Patent Citations
Multi-agent reinforcement learning cooperation method based on dynamic graph communication
CN116306966A
Cloud boundary network active decision defense method based on deep reinforcement learning
CN116599704A
Multi-intersection traffic signal cooperative control method
CN117133138A
Space-air-ground integrated slice network resource allocation method
CN117750505A
Multi-agent traffic area signal control method based on digital twinning
CN118097989A
Cited By
Multi-starting-point sequence decision reinforcement learning method based on dynamic mask attention
CN121119029A
