Multi-agent reinforcement learning method for fair adaptive traffic signal control

Abstract

The application discloses a kind of multi-agent reinforcement learning methods for fair adaptive traffic signal control, comprising: the problem of fair adaptive traffic signal control is modeled as constraint Markov game, the goal is to find an optimal joint strategy can maximize the average expected return of all agents while meeting the cost constraints of each agent;On the basis of the constraint Markov game, the confidence region policy optimization agent function is used to optimize the cumulative discounted reward and discounted cost in the average expected return, and the Lagrange multiplier method is used to convert the constrained problem into an unconstrained problem;Vehicle fairness multi-agent approximate policy optimization is used to learn the cooperative control strategy that maximizes the overall return of the unconstrained problem under the waiting time constraint.The application considers both efficiency and fairness when controlling traffic signals at multiple intersections.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a multi-agent reinforcement learning method for fair adaptive traffic signal control. Background Technology

[0002] Traditional traffic signal control methods rely on pre-set traffic signals and cannot take into account real-time dynamic traffic flow, resulting in low traffic control efficiency. Adaptive traffic signal control, on the other hand, dynamically adjusts signal timing schemes based on real-time traffic data, thereby improving traffic efficiency and safety. It is also an important direction for artificial intelligence and big data to further apply current traffic data, in addition to traffic data prediction.

[0003] In recent years, the application of deep reinforcement learning (RL) algorithms to adaptive traffic signal control has gradually become a research hotspot, proving to be effective and promising. In these applications, the traffic lights at each intersection are controlled by one or more control units trained using reinforcement learning algorithms. In the field of reinforcement learning, these control units are usually referred to as agents. The reinforcement learning agent collects state information such as queue length, car position, and vehicle speed. Then, the agent takes actions according to a policy and observes the impact of these actions on the traffic environment. Based on this, it gradually improves the policy to try to find the optimal policy that achieves the goal. Existing RL algorithms can be broadly classified into two categories: 1) RL algorithms based on Deep Q-Network (DQN), which use a neural network to determine the actions taken by an RL agent; and 2) RL algorithms based on Actor-Critic networks, which use an additional neural network to estimate state advantages.

[0004] However, most of these algorithms focus on maximizing throughput or minimizing average vehicle travel time to improve overall road network flow, without considering that achieving this goal might sacrifice the interests of certain lanes. This could lead to vehicles in lanes with lower traffic volume experiencing endless waits, creating unfairness. This is unreasonable and results in a terrible driving experience for drivers. Therefore, in real-world scenarios like traffic signal control, it's necessary to consider not only maximizing flow but also fairness. Existing work that simultaneously considers both flow and fairness in traffic signal control is scarce and mainly limited to isolated intersections or very simple composite scenarios. Therefore, there is an urgent need to develop an effective method that balances efficiency and fairness. Summary of the Invention

[0005] In view of this, this invention provides a multi-agent reinforcement learning method for fair adaptive traffic signal control. This invention proposes a multi-agent reinforcement learning framework, namely Vehicle Fairness Multi-agent Proximal Policy Optimization (VF-MAPPO), for training the proposed optimal ATSC control strategy for multi-intersection traffic signal control oriented towards vehicle fairness. This approach balances efficiency and fairness when performing traffic signal control at multi-intersections.

[0006] Therefore, the present invention provides the following technical solution:

[0007] This invention provides a multi-agent reinforcement learning method for fair adaptive traffic signal control, the method comprising:

[0008] The problem of fair adaptive traffic signal control is modeled as a constrained Markov game, with the goal of finding an optimal joint strategy that maximizes the average expected return of all agents while satisfying the cost constraints of each agent.

[0009] Based on constrained Markov games, we use confidence region strategy optimization agent function to optimize the cumulative discounted reward and discounted cost in the average expected return, and use the Lagrange multiplier method to transform the constrained problem into an unconstrained problem.

[0010] We employ a multi-agent approximation strategy optimization based on vehicle fairness to learn a cooperative control strategy that maximizes the overall reward in an unconstrained problem while satisfying the waiting time constraint.

[0011] Furthermore, each intersection is considered as an intelligent agent, with its state consisting of the current phase, the number of vehicles, and the waiting time of the first vehicle. The reward is defined as the sum of the number of waiting vehicles in all approach lanes, and the cost is defined as the waiting time of all lanes. The optimization objective is defined as follows:

[0012]

[0013]

[0014] Among them, s t The state of the environment at time step t. It is the average expected return of all intelligent agents. Cost constraints for each agent i.

[0015] Furthermore, the confidence domain policy optimization proxy function used to optimize cumulative discount rewards includes:

[0016]

[0017] Where, π and These are the current policy and the next candidate policy, where s is the state and a is the next candidate policy. i It is the action of agent i, ρ π To reduce the frequency of visits, It is the average expected return of agent i. It is a reward advantage function.

[0018] Furthermore, the confidence region policy optimization surrogate function used for the discount cost function includes:

[0019]

[0020] Where, π and These are the current policy and the next candidate policy, respectively. It is the average expected cost of agent i. It is a cost advantage function.

[0021] Furthermore, combining the idea of ​​importance sampling, the optimization objective is modified to:

[0022]

[0023] in, n is the number of agents, and θ is the neural network parameter for policy π. It calculates the KL divergence, where δ is a very small constant.

[0024] Furthermore, using the Lagrange multiplier method, the constrained objective is transformed into an unconstrained objective:

[0025]

[0026] in, λ is the Lagrange multiplier, and l is the cost constraint value.

[0027] Furthermore, it also includes: approximating the error of the KL divergence using a clipping operation, with the objective in the form of:

[0028]

[0029] Here, clip is the clipping operation, and ε is the clipping ratio.

[0030] Furthermore, a vehicle fairness multi-agent approximation strategy optimization is adopted to learn a cooperative control strategy that maximizes the overall reward while satisfying the waiting time constraint, including:

[0031] In each round:

[0032] Initialize the experience pool;

[0033] At each time step, each agent obtains the distribution of actions according to the policy, selects actions based on the action distribution, and predicts the reward and cost values ​​of the current state.

[0034] After all time steps have been executed, perform the joint action to obtain the reward, the next state, and the next observation.

[0035] Store experiences in an experience pool; experiences include: at time step t, the environment is in state s. ( t ) The agent's observations ( t ) Joint action a ( t ) The next state s ( t +1) The next observation s ( t +1) Reward r ( t ) Cost c ( t ) ;

[0036] Calculate the cumulative discount reward and cumulative discount cost for each agent;

[0037] The generalized advantage estimation method is used to calculate the reward advantage function and cost advantage function for each agent;

[0038] In each iteration, data is randomly sampled from the experience pool;

[0039] minimize Update actor network; update Lagrange multipliers; minimize

[0040] Update the critic network.

[0041] Advantages and positive effects of this invention: To solve the fair-oriented adaptive traffic signal control (ATSC) problem, this invention models the problem as a constrained Markov game (CMG), a special type of Markov game that requires not only maximizing rewards but also adhering to specified constraints. Based on CMG, the optimization objective is transformed using a Trust Region Policy Optimization (TRPO) surrogate function. The constrained problem is transformed into an unconstrained problem using the Lagrange multiplier method. Furthermore, a multi-agent cooperative method extended from Mappo, named VF-MAPPO, is proposed to learn a cooperative control strategy that maximizes the overall reward while satisfying waiting time constraints. Experimental results show that, among almost all commonly used evaluation metrics for ATSC, the algorithm provided by this invention guarantees comparable flowability compared to existing methods; and among fairness evaluation metrics, this algorithm exhibits the best fairness compared to existing algorithms. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the multi-agent reinforcement learning framework in an embodiment of the present invention;

[0044] Figure 2 These are schematic diagrams of three traffic scenarios in embodiments of the present invention;

[0045] Figure 3 This is the reward graph for all algorithms under three traffic scenarios in this embodiment of the invention;

[0046] Figure 4 This is a cost diagram of all algorithms under three traffic scenarios in this embodiment of the invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0048] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] Considering the different mathematical properties of the research objectives, existing methods that obtain comprehensive requirements by directly optimizing the weighted sum of multiple objectives still have considerable room for improvement. Unlike most existing research that models traffic signal control as an unconstrained optimization problem, this invention proposes to model the optimization objective as a constrained optimization problem and use the Lagrange transform to convert the constrained optimization problem into an unconstrained optimization problem.

[0050] First, the fair MARL problem is modeled as a partially observable CMG, using G =<N,S,O,A,P,R,C,c,γ> The CMG extends the standard Markov Game (MG) by adding a constraint set C, denoted by G =<N,S,O,A,P,R,C,l,γ> Let N = {1, ..., n} be the set of agents; S be the state space; and O be the observation space, i.e., O = ... <O1,...,O n >, where O i A represents the local observations obtained by agent i; A represents the joint action space of all agents, i.e., A = <a1,...,a n >, where A i P is the local action space of agent i; P:S×A×S→[0,1] is the state transition function; R is the joint reward function, expressed as R= <r1,...,r n >, where r i C is the individual reward of agent i; C is the joint reward function, expressed as C = <c1,...,c n >, where c i Let be the individual cost of agent i; l be the cost constraint value; and γ ∈ [0,1) be the discount factor. At time step t, the environment is in state s. t Each agent i observes its own observations. And according to its strategy Get Action Execute joint action a t Receive joint reward r t Simultaneous payment of joint costs c t The environment transitions to the next state s t+1 The goal is to find an optimal joint strategy π = <π1,...,π n >Can maximize the average expected return of all agents Simultaneously satisfying the cost constraints of each agent i

[0051] Specifically, in this invention, each intersection is considered as an intelligent agent, and its state consists of the current phase, the number of vehicles, and the waiting time of the first vehicle. The reward is defined as the sum of the number of waiting vehicles in all approach lanes, and the cost is defined as the waiting time of all lanes. The optimization objective is defined as follows:

[0052]

[0053] The TRPO method introduces a proxy function to optimize cumulative discount rewards. The proxy function is defined as follows:

[0054]

[0055] Where π and These are the current strategy and the next candidate strategy, respectively.

[0056] Similarly, the surrogate function for the discount cost function can be defined as follows:

[0057]

[0058] Combining the idea of ​​importance sampling, formula (1) can be rewritten as:

[0059]

[0060] in,

[0061] Next, using the Lagrange multiplier method, the constrained objective is transformed into an unconstrained objective, and formula (2) is transformed into:

[0062]

[0063] in,

[0064] Finally, the error of the KL divergence is approximated using the clip operation, and the objective is in the form of:

[0065]

[0066] By optimizing formula (4), we can achieve the goal of maximizing rewards while meeting cost constraints, which means we can balance the efficiency and fairness of traffic adaptive signal control.

[0067] The VF-MAPPO framework is as follows Figure 1 As shown, the pseudocode is shown in Table 1.

[0068] Table 1

[0069]

[0070]

[0071] This invention is applied to traffic signal control tasks, aiming to design a multi-agent reinforcement learning framework for fair traffic signal control. Simulation experiments were conducted on a simulator using both real and synthetic traffic environment data to verify the performance of the model and algorithm. The experimental procedure is as follows:

[0072] The method of the present invention was evaluated using three available public transportation networks on Cityflow (a popular traffic simulator): two real traffic networks (Jinan 3×4 and New York 16×3) and one 4×4 synthetic traffic network. Figure 2 As shown, the left image shows a partial transportation network in Jinan, the middle image shows a partial transportation network in New York, and the right image shows a composite transportation network.

[0073] The intersections in all three scenarios are homogeneous and have the same action space.

[0074] (1) Comparative Experiment

[0075] VF-MAPPO is compared with two traditional traffic signal control methods (FixedTime and MaxPressure) and four state-of-the-art reinforcement learning methods (DGN, PNC-HDQN, PressureLight, and MA2C).

[0076] from Figure 3 and Figure 4 As can be seen, the VF-MAPPO algorithm of this invention achieves the lowest cost on all datasets compared to other algorithms, and is particularly effective on the Jinan and New York datasets. Figure 3 (a) is the reward graph for the Jinan dataset. Figure 3 (b) is the reward graph for the New York dataset. Figure 3 (c) is the reward map of the synthetic dataset. Figure 4 (a) is the cost diagram of the Jinan dataset. Figure 4 (b) is a cost graph of the New York dataset. Figure 4(c) The best reward was achieved on the cost graph of the synthetic dataset, and it was not significantly different from the best-performing DGN algorithm on the synthetic dataset. The training performance comparison is shown in Table 2.

[0077] Table 2

[0078]

[0079]

[0080] Table 1 compares the flowability and fairness metrics of all algorithms. Among all road networks, the VF-MAPPO algorithm in this invention has the best longest waiting time in terms of fairness, and its flowability is also almost the best.

[0081] Experimental results show that, among almost all evaluation metrics commonly used in ATSC, VF-MAPPO, compared to existing methods, not only reduces the longest waiting time for vehicles and ensures fairness, but also guarantees overall performance.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning method for fair adaptive traffic signal control, characterized in that, The method includes: The problem of fair adaptive traffic signal control is modeled as a constrained Markov game, with the goal of finding an optimal joint strategy that maximizes the average expected return of all agents while satisfying the cost constraints of each agent. Based on the constrained Markov game, the confidence region strategy optimization agent function is used to optimize the cumulative discounted reward and discounted cost in the average expected return, and the constrained problem is transformed into an unconstrained problem by using the Lagrange multiplier method. A multi-agent approximation strategy for vehicle fairness is adopted to learn a cooperative control strategy that maximizes the overall reward under the waiting time constraint in the unconstrained problem. The confidence region policy optimization proxy function used to optimize cumulative discount rewards includes: ； wherein, and are the current policy and the next candidate policy, respectively, s is the state, is the action of the agent , is the discounted visit frequency, is the average expected return of the agent , is the reward advantage function; The confidence region policy optimization surrogate function used for the discount cost function includes: ； where and are the current policy and the next candidate policy, respectively, is the agent 's average expected cost, is the cost advantage function.

2. The multi-agent reinforcement learning method for fair adaptive traffic signal control according to claim 1, wherein, Each intersection is treated as an intelligent agent, with its state consisting of the current phase, the number of vehicles, and the waiting time of the first vehicle. The reward is defined as the sum of the number of waiting vehicles in all approach lanes, and the cost is defined as the waiting time of all lanes. The optimization objective is defined as follows: ； ； where, is the time step t the state of the environment, is the average expected return of all agents, is the cost constraint for each agent .

3. The multi-agent reinforcement learning method for fair adaptive traffic signal control according to claim 2, characterized in that, Combining the concept of importance sampling, the optimization objective is modified to: ； in, n is the number of agents. It is a strategy Neural network parameters, It calculates the KL divergence. It is a very small constant.

4. The multi-agent reinforcement learning method for fair adaptive traffic signal control according to claim 3, characterized in that, Using the Lagrange multiplier method, the constrained objective is transformed into an unconstrained objective: ； in, , It is a Lagrange multiplier. l It is a cost constraint value.

5. The multi-agent reinforcement learning method for fair adaptive traffic signal control according to claim 4, characterized in that, Also includes: The error of the KL divergence is approximated using a clipping operation, and the objective is in the form of: ； Clip is the editing operation. It's the editing ratio.

6. The multi-agent reinforcement learning method for fair adaptive traffic signal control according to claim 5, characterized in that, A vehicle fairness multi-agent approximation strategy optimization is adopted to learn a cooperative control strategy that maximizes the overall reward while satisfying waiting time constraints, including: In each round: Initialize the experience pool; At each time step, each agent obtains the distribution of actions according to the policy, selects actions based on the action distribution, and predicts the reward and cost values ​​of the current state. After all time steps have been executed, perform the joint action to obtain the reward, the next state, and the next observation. Experience is stored in an experience pool; the experience includes: at time step The environment is in a state Observation of intelligent agents joint action Next state Next observation ,award cost ; Calculate the cumulative discount reward and cumulative discount cost for each agent; The generalized advantage estimation method is used to calculate the reward advantage function and cost advantage function for each agent; In each iteration, data is randomly sampled from the experience pool; minimize Update actor network; update Lagrange multipliers; minimize Update the critic network.

Images