A multi-agent road traffic signal control method based on a deep reinforcement learning algorithm

Abstract

A road traffic signal control optimization method based on deep reinforcement learning algorithm includes: confirming the state information of the intersection based on road network environment information and current traffic flow information; proposing multiple action spaces based on the original MUDQN algorithm, executing actions in different action spaces under different conditions, and proposing a new state space and a new reward function. The model is trained using data stored in an experience replay pool, enabling multiple agents to eventually reach a NASH equilibrium state. The input parameters of this traffic signal control optimization method are the intersection observation information corresponding to the traffic lights and the currently obtained reward values ​​of adjacent intersections, and the output parameter of the traffic signal control model module is the phase of the traffic lights at the current time.

Description

Technical Field

[0001] This invention relates to a traffic signal control method, and more particularly to a multi-agent road traffic signal control method based on a deep reinforcement learning algorithm. It belongs to the field of intelligent transportation vehicle artificial intelligence control technology. Background Technology

[0002] In recent years, traffic congestion has wasted travel time, exacerbated exhaust emissions, and caused serious economic losses. Controlling traffic signals can alleviate traffic congestion to some extent. Currently, with the rapid development of reinforcement learning, researchers have found it to be the most suitable framework for solving traffic signal control problems and have begun to study reinforcement learning techniques to achieve cooperation among traffic signals. In particular, in large-scale road networks, cooperation between intersections is crucial for alleviating traffic congestion. Summary of the Invention

[0003] A multi-agent road traffic signal control method based on deep reinforcement learning algorithm is proposed to address the shortcomings of urban traffic signal control schemes. This method fully utilizes urban road resources, reduces potential traffic congestion, and improves the efficiency of vehicle passage through intersections and the safety and comfort of passengers.

[0004] The technical solution of this invention is implemented as follows:

[0005] A multi-agent road traffic signal control method based on deep reinforcement learning algorithm is characterized by using the simulation software SUMO to create multiple different intersection road environments, designing different action spaces, state spaces, and reward functions according to different intersections, constructing a multi-agent Q-network based on the deep Q-learning algorithm, training it until convergence and Nash equilibrium is achieved among the agents, and finally interacting with the environment through the Traci interface in SUMO. The method includes the following steps:

[0006] Step 1: Construct a directed graph for the traffic network controlled by the large-scale traffic signal system. The nodes of the directed graph represent traffic intersections, and the edges represent roads. The large-scale traffic signal control system model includes multiple agents, each of which corresponds one-to-one with a traffic intersection in the traffic network. Collect information on road intersections and traffic flow. Based on the above data, use simulation software to simulate the traffic artery and restore the real vehicle convergence information at road intersections.

[0007] Step 2: Obtain intersection environment information from the simulation environment. Based on the obtained environment information, set the state space of the deep reinforcement learning model. The state space serves as the input to the deep reinforcement learning model. The state space of the deep reinforcement learning model consists of the states of each traffic intersection controlled by the large-scale traffic signal system. The state space O of the model is represented as:

[0008]

[0009] in, Indicates the current action A of agent i. Let F represent the number of waiting vehicles in the F traffic flow directions for agent i. This represents the number of waiting vehicles in the F* traffic flow directions that are ready to travel to agent i from its neighboring agents. It represents the current Q value of the R neighboring agents of agent i, also known as the expected reward value;

[0010] Step 3: Set up the action space of the deep reinforcement learning model. The model's action space consists of the actions of various traffic intersections controlled by a large-scale traffic signal system. Different agents have different action spaces, and each agent has two action spaces, A and A*. The execution durations of the two action spaces A and A* of the same agent are different to cope with the changing traffic flow at different times. When the number of vehicles waiting at the intersection is greater than the intersection traffic flow threshold, action space A is used; otherwise, action space A* is used. The action space of agent i at a four-phase intersection is represented as follows:

[0011] A i / A i * = [a1, a2, a3, a4],

[0012] Among them, [a1,a2,a3,a4] are [WET,NST,WEL,NSL] respectively. WET represents straight east-west traffic, NST represents straight north-south traffic, WEL represents left east-west traffic, and NSL represents left north-south traffic.

[0013] The action space of the two-phase gate merging agent j is represented as follows:

[0014] A j / A j * = [a1, a2]

[0015] Where [a1, a2] are [P, S] respectively, where P means that passage is allowed and S means that passage is not allowed;

[0016] When the action performed by the intelligent agent changes, i.e. when the green light lane changes, the yellow light phase needs to be inserted to mitigate vehicle safety issues caused by sudden changes in the green light phase.

[0017] Step 4: Set the reward function for the deep reinforcement learning model. When the intersection takes a corresponding action based on the model's output, the action is scored (reward value) based on the changing environmental information. Different agents operate in different environments, so the reward function will also be different.

[0018] (1) The R1 function refers to the reward function for a crossroads or T-junction:

[0019] R1=ω1*r people +ω2*r vehicle

[0020] Where ω1 and ω2 correspond to weights, ω1 + ω2 = 1; r people =∑ k w t (k)-∑ k w t-1 (k) represents the difference between the total waiting time of all pedestrians at time t and the total waiting time of all pedestrians on the road at time t-1, where k represents the total number of crosswalks at the intersection; r vehicle =∑ l p t (l)-∑ l p t-1 (l) represents the difference between the total time when all vehicles on the road have a speed of 0 at time t and the total time when all vehicles on the road have a speed of 0 at time t-1, and l represents all the lanes entering the intersection;

[0021] (2) Reward value function R2 for gate merging:

[0022] R2=μ1*r safety +μ2*r com

[0023] Where, r safety For safe sub-rewards, r com The comfort sub-reward, μ1 and μ2 are the weights of the corresponding sub-rewards, and μ1 + μ2 = 1;

[0024] r safety The safety sub-reward makes its first judgment based on the current agent's action. When the agent performs action S, the safety sub-reward r... safety =0; When the agent performs action P, a second judgment is made based on the distance between vehicles. When the distance between vehicles is less than the minimum safe distance, the safety sub-reward r is given. safety =-100, when the vehicle spacing is greater than the minimum safe spacing, the comfort sub-reward Where d i,e Indicates the longitudinal position of the vehicle, d i,f d i,h s represents the longitudinal position of the vehicles in front of and behind the main lane, respectively. i,e The longitudinal velocity of the vehicle is represented by s. i,f s i,h These represent the longitudinal speeds of vehicles in front of and behind the main lane, respectively.

[0025] r comComfort Sub-Rewards

[0026] Where r wait This refers to the current waiting time of the vehicle, r dif This represents the difference between the current vehicle's speed and the average speed of vehicles ahead and behind in the main lane. These are the weights of the corresponding sub-rewards.

[0027] Step 5: Initialize the experience replay pool. Obtain the states, actions in the current state, expected reward values ​​of actions, and the next state after completing the action of multiple agents controlled by the large-scale traffic signal system and store them in the experience replay pool. When the data storage in the experience replay pool is greater than the minimum storage capacity, extract a batch of stored data for neural network training. Each batch of data extracted for training is called a set. Set the target number of training sets to be extracted.

[0028] Step Six: Construct a deep neural network to fit the state-action value function. The input to the neural network is a state space composed of state information obtained from the simulation environment. The output of the neural network is the Q-value of each action in the input state, i.e., the expected reward value of each action in the current state. The agent uses the Q-learning equation to update the Q-value. The Q-learning equation is:

[0029] Q(s,a)=r t +γ*max a Q(O t+1 ,a)a∈A / A*

[0030] Where r t The reward for the current action is γ, and the loss factor is maxQ(O). t+1 a) indicates that state O t+1 The input is a neural network, and the maximum Q value is output. Data is extracted from the experience replay pool and stored. The neural network is trained using the Q value output by the neural network and the updated Q value through the mean squared error loss function until the number of training sets reaches the required target number of training sets, so that the NSAH equilibrium is achieved among the agents and the target network converges.

[0031] Step 7: Obtain real-time vehicle status information generated by SUMO simulation software through the Traci interface and input it into the trained multi-agent road traffic signal control model. The corresponding decision behavior is obtained for each intersection and returned to SUMO to execute the corresponding phase.

[0032] Compared with the prior art, the advantages of the present invention are obvious, mainly reflected in:

[0033] 1. This invention proposes a multi-action space method. When the environment in which the agent is located or a certain environmental condition reaches a certain threshold, the current action space is changed to make the current action space more consistent with the current environment.

[0034] 2. This invention defines a new method for representing the state of intersections, which is no longer limited to the traditional single-agent intersection state representation method. It incorporates the expected number of vehicles entering the intersection and the reward value of adjacent intersections, which can effectively improve the connection between multiple intersections.

[0035] 3. This invention defines a new reward function and uses different reward functions for different intersection scenarios. For intersections or T-junctions, the reward function includes vehicle waiting time and pedestrian waiting time, taking pedestrian comfort into account while considering the rapid passage of vehicles. For merging ramps, vehicle safety and passenger comfort are taken into account. This avoids the problem of traditional reinforcement learning overemphasizing road conditions while neglecting pedestrian and passenger comfort. Attached Figure Description

[0036] This invention has a total of appendices Figure 2 width.

[0037] Figure 1 This is a schematic diagram of the road network in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of the control method of the present invention. Detailed Implementation

[0039] like Figure 1 , 2 The invention discloses a multi-agent road traffic signal control method based on deep reinforcement learning algorithm. Its features include: creating multiple different intersection road environments using the simulation software SUMO; designing different action spaces, state spaces, and reward functions for different intersections; constructing a multi-agent Q-network based on the deep Q-learning algorithm; training the network until convergence and Nash equilibrium is achieved among the agents; and finally, interacting with the environment through the Traci interface in SUMO. The method includes the following steps:

[0040] Step 1: Construct a directed graph for the traffic network controlled by the large-scale traffic signal system. The nodes of the directed graph represent traffic intersections, and the edges represent roads. The large-scale traffic signal control system model includes multiple agents, each of which corresponds one-to-one with a traffic intersection in the traffic network. Collect information on road intersections and traffic flow. Based on the above data, use simulation software to simulate the traffic artery and restore the real vehicle convergence information at road intersections.

[0041] Step 2: Obtain intersection environment information from the simulation environment. Based on the obtained environment information, set the state space of the deep reinforcement learning model. The state space serves as the input to the deep reinforcement learning model. The state space of the deep reinforcement learning model consists of the states of each traffic intersection controlled by the large-scale traffic signal system. The state space O of the model is represented as:

[0042]

[0043] in, Indicates the current action A of agent i. Let F represent the number of waiting vehicles in the F traffic flow directions for agent i. This represents the number of waiting vehicles in the F* traffic flow directions that are ready to travel to agent i from its neighboring agents. It represents the current Q value of the R neighboring agents of agent i, also known as the expected reward value;

[0044] Step 3: Set up the action space of the deep reinforcement learning model. The model's action space consists of the actions of various traffic intersections controlled by a large-scale traffic signal system. Different agents have different action spaces, and each agent has two action spaces, A and A*. The execution durations of the two action spaces A and A* of the same agent are different to cope with the changing traffic flow at different times. When the number of vehicles waiting at the intersection is greater than the intersection traffic flow threshold, action space A is used; otherwise, action space A* is used. The action space of agent i at a four-phase intersection is represented as follows:

[0045] A i / A i * = [a1, a2, a3, a4],

[0046] Among them, [a1,a2,a3,a4] are [WET,NST,WEL,NSL] respectively. WET represents straight east-west traffic, NST represents straight north-south traffic, WEL represents left east-west traffic, and NSL represents left north-south traffic.

[0047] The action space of the two-phase gate merging agent j is represented as follows:

[0048] A j / A j * = [a1, a2]

[0049] Where [a1, a2] are [P, S] respectively, where P means that passage is allowed and S means that passage is not allowed;

[0050] When the action performed by the intelligent agent changes, i.e. when the green light lane changes, the yellow light phase needs to be inserted to mitigate vehicle safety issues caused by sudden changes in the green light phase.

[0051] Step 4: Set the reward function for the deep reinforcement learning model. When the intersection takes a corresponding action based on the model's output, the action is scored (reward value) based on the changing environmental information. Different agents operate in different environments, so the reward function will also be different.

[0052] (1) The R1 function refers to the reward function for a crossroads or T-junction:

[0053] R1=ω1*r people +ω2*r vehicle

[0054] Where ω1 and ω2 correspond to weights, ω1 + ω2 = 1; r people =∑ k w t (k)-∑ k w t-1 (k) represents the difference between the total waiting time of all pedestrians at time t and the total waiting time of all pedestrians on the road at time t-1, where k represents the total number of crosswalks at the intersection; r vehicle =∑ l p t (l)-∑ l p t-1 (l) represents the difference between the total time when all vehicles on the road have a speed of 0 at time t and the total time when all vehicles on the road have a speed of 0 at time t-1, and l represents all the lanes entering the intersection;

[0055] (2) Reward value function R2 for gate merging:

[0056] R2=μ1*r safety +μ2*r com

[0057] Where, r safety For safe sub-rewards, r com The comfort sub-reward, μ1 and μ2 are the weights of the corresponding sub-rewards, and μ1 + μ2 = 1;

[0058] r safety The safety sub-reward makes its first judgment based on the current agent's action. When the agent performs action S, the safety sub-reward r... safety =0; When the agent performs action P, a second judgment is made based on the distance between vehicles. When the distance between vehicles is less than the minimum safe distance, the safety sub-reward r is given. safety =-100, when the vehicle spacing is greater than the minimum safe spacing, the comfort sub-reward Where d i,e Indicates the longitudinal position of the vehicle, d i,f d i,h s represents the longitudinal position of the vehicles in front of and behind the main lane, respectively. i,e The longitudinal velocity of the vehicle is represented by s. i,f s i,h These represent the longitudinal speeds of vehicles in front of and behind the main lane, respectively.

[0059] r com Comfort Sub-Rewards

[0060] Where r wait This refers to the current waiting time of the vehicle, r dif This represents the difference between the current vehicle's speed and the average speed of vehicles ahead and behind in the main lane. These are the weights of the corresponding sub-rewards.

[0061] Step 5: Initialize the experience replay pool. Obtain the states, actions in the current state, expected reward values ​​of actions, and the next state after completing the action of multiple agents controlled by the large-scale traffic signal system and store them in the experience replay pool. When the data storage in the experience replay pool is greater than the minimum storage capacity, extract a batch of stored data for neural network training. Each batch of data extracted for training is called a set. Set the target number of training sets to be extracted.

[0062] Step Six: Construct a deep neural network to fit the state-action value function. The input to the neural network is a state space composed of state information obtained from the simulation environment. The output of the neural network is the Q-value of each action in the input state, i.e., the expected reward value of each action in the current state. The agent uses the Q-learning equation to update the Q-value. The Q-learning equation is:

[0063] Q(s,a)=r t +γ*max a Q(O t+1 ,a)a∈A / A*

[0064] Where r t The reward for the current action is γ, and the loss factor is maxQ(O). t+1 a) indicates that state O t+1 The input is a neural network, and the maximum Q value is output. Data is extracted from the experience replay pool and stored. The neural network is trained using the Q value output by the neural network and the updated Q value through the mean squared error loss function until the number of training sets reaches the required target number of training sets, so that the NSAH equilibrium is achieved among the agents and the target network converges.

[0065] Step 7: Obtain real-time vehicle status information generated by SUMO simulation software through the Traci interface and input it into the trained multi-agent road traffic signal control model. The corresponding decision behavior is obtained for each intersection and returned to SUMO to execute the corresponding phase.

Claims

1. A multi-agent road traffic signal control method based on deep reinforcement learning algorithm, characterized in that, The method involves creating multiple road environments with different intersections using the SUMO simulation software, designing different action spaces, state spaces, and reward functions for each intersection, constructing a multi-agent Q-network based on the deep Q-learning algorithm, training it until convergence and Nash equilibrium is achieved among the agents, and finally interacting with the environment through the Traci interface in SUMO. The steps include: Step 1: Construct a directed graph for the traffic network controlled by the large-scale traffic signal system. The nodes of the directed graph represent traffic intersections, and the edges represent roads. The large-scale traffic signal control system model includes multiple agents, each of which corresponds one-to-one with a traffic intersection in the traffic network. Collect information on road intersections and traffic flow. Based on the above data, use simulation software to simulate the traffic artery and restore the real vehicle convergence information at road intersections. Step 2: Obtain intersection environment information from the simulation environment. Based on the obtained environment information, set the state space of the deep reinforcement learning model. The state space serves as the input to the deep reinforcement learning model. The state space of the deep reinforcement learning model consists of the states of each traffic intersection controlled by the large-scale traffic signal system. The state space O of the model is represented as: in, Indicates the current action A of agent i. Let F represent the number of waiting vehicles in the F traffic flow directions for agent i. This represents the number of waiting vehicles in the F* traffic flow directions that are ready to travel to agent i from its neighboring agents. It represents the current Q value of the R neighboring agents of agent i, also known as the expected reward value; Step 3: Set up the action space of the deep reinforcement learning model. The model's action space consists of the actions of various traffic intersections controlled by a large-scale traffic signal system. Different agents have different action spaces, and each agent has two action spaces, A and A*. The execution durations of the two action spaces A and A* of the same agent are different to cope with the changing traffic flow at different times. When the number of vehicles waiting at the intersection is greater than the intersection traffic flow threshold, action space A is used; otherwise, action space A* is used. The action space of agent i at a four-phase intersection is represented as follows: A i / A i *=[a1,a2,a3,a4], Among them, [a1,a2,a3,a4] are [WET,NST,WEL,NSL] respectively. WET represents straight east-west traffic, NST represents straight north-south traffic, WEL represents left east-west traffic, and NSL represents left north-south traffic. The action space of the two-phase gate merging agent j is represented as follows: A j / A j *=[a1,a2] Where [a1, a2] are [P, S] respectively, where P means that passage is allowed and S means that passage is not allowed; When the action performed by the intelligent agent changes, i.e. when the green light lane changes, the yellow light phase needs to be inserted to mitigate vehicle safety issues caused by sudden changes in the green light phase. Step 4: Set the reward function for the deep reinforcement learning model. When the intersection takes a corresponding action based on the model's output, the action is scored (reward value) based on the changing environmental information. Different agents operate in different environments, so the reward function will also be different. (1) The R1 function refers to the reward function for a crossroads or T-junction: R1=ω1*r people +ω2*r vehicle Where ω1 and ω2 correspond to weights, ω1 + ω2 = 1; r people =∑ k w t (k)-∑ k w t-1 (k) represents the difference between the total waiting time of all pedestrians at time t and the total waiting time of all pedestrians on the road at time t-1, where k represents the total number of crosswalks at the intersection; r vehicle =∑ l p t (l)-∑ l p t-1 (l) represents the difference between the total time when all vehicles on the road have a speed of 0 at time t and the total time when all vehicles on the road have a speed of 0 at time t-1, and l represents all the lanes entering the intersection; (2) Reward function R2 for gate merging: R2=μ1*r safety +μ2*r com Where, r safety For safe sub-rewards, r com The comfort sub-reward, μ1 and μ2 are the weights of the corresponding sub-rewards, and μ1 + μ2 = 1; r safety The safety sub-reward makes its first judgment based on the current agent's action. When the agent performs action S, the safety sub-reward r... safety =0; When the agent performs action P, a second judgment is made based on the distance between vehicles. When the distance between vehicles is less than the minimum safe distance, the safety sub-reward r is given. safety =-100, when the vehicle spacing is greater than the minimum safe spacing, the safety sub-reward is... Where d i,e Indicates the longitudinal position of the vehicle, d i,f d i,h s represents the longitudinal position of the vehicles in front of and behind the main lane, respectively. i,e The longitudinal velocity of the vehicle is represented by s. i,f s i,h These represent the longitudinal speeds of vehicles in front of and behind the main lane, respectively. r com Comfort Sub-Rewards Where r wait This refers to the current waiting time of the vehicle, r dif This represents the difference between the current vehicle's speed and the average speed of vehicles in front and behind it in the main lane. These are the weights of the corresponding sub-rewards. Step 5: Initialize the experience replay pool. Obtain the states, actions in the current state, expected reward values ​​of actions, and the next state after completing the action of multiple agents controlled by the large-scale traffic signal system and store them in the experience replay pool. When the data storage in the experience replay pool is greater than the minimum storage capacity, extract a batch of stored data for neural network training. Each batch of data extracted for training is called a set. Set the target number of training sets to be extracted. Step Six: Construct a deep neural network to fit the state-action value function. The input to the neural network is a state space composed of state information obtained from the simulation environment. The output of the neural network is the Q-value of each action in the input state, i.e., the expected reward value of each action in the current state. The agent uses the Q-learning equation to update the Q-value. The Q-learning equation is: Q(s,a)=r t +γ*maxQ(O t+1 ,a)a∈A / A* Where r t The reward for the current action is γ, and the loss factor is maxQ(O). t+1 a) indicates that state O t+1 The input is a neural network, and the maximum Q value is output. Data is extracted from the experience replay pool and stored. The neural network is trained using the Q value output by the neural network and the updated Q value through the mean squared error loss function until the number of training sets reaches the required target number of training sets, so that the NSAH equilibrium is achieved among the agents and the target network converges. Step 7: Obtain real-time vehicle status information generated by SUMO simulation software through the Traci interface and input it into the trained multi-agent road traffic signal control model. The corresponding decision behavior is obtained for each intersection and returned to SUMO to execute the corresponding phase.

Images