Coordinated control method, device and readable storage medium for traffic lights at multiple intersections

Abstract

The present invention relates to a method and device for collaborative control of traffic lights at multiple intersections, and a readable storage medium. The method comprises: obtaining traffic flow information, inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm, so as to obtain and store the state, action and reward function of each agent; using an interpretable influence mechanism based on an efficient link neural network to obtain the importance coefficient of input data on different traffic networks, and to obtain the weighted aggregated traffic flow hidden variables; using a biased ReLU neural network to approximate the value function and strategy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework; using a centralized training distributed execution method, the actor of each agent obtains its own strategy function through traffic flow information training; the centralized critic is trained to obtain a joint value function based on the traffic flow hidden variables, and obtains the optimal solution of the multi-agent reinforcement learning algorithm. The present invention can improve the efficiency and safety of road traffic flow.

Description

Technical Field

[0001] The present invention relates to a method, device and readable storage medium for coordinated control of traffic lights at multiple intersections, and belongs to the technical field of intelligent traffic light control. Background Art

[0002] Transportation is a key driving force for economic and social development and one of the manifestations of urban economic competitiveness. The development of transportation infrastructure has brought many conveniences to people's lives, but it has also brought a series of problems. The most common problem in urban transportation is congestion, which brings many inconveniences to people's travel, increases driving risks, reduces logistics efficiency, causes energy waste, causes overall economic losses to society, and brings huge pressure on the environment.

[0003] Traffic lights can adjust the traffic flow on different roads by controlling the phases of red and green lights in different directions at intersections, thereby achieving the purpose of traffic diversion, reducing congestion, and reducing the waiting time of vehicles at intersections. Generally speaking, traffic light control methods are mainly divided into three types: timing control, sensing control, and adaptive control. Among them, timing control updates the phase of intersection lights based on a predetermined schedule, and sensing control uses sensors to obtain the current traffic flow status and uses predetermined rules to set the green light phase of the intersection. Both methods do not take into account long-term traffic conditions, cannot optimize signal light control in real time, and lack the ability to dynamically alleviate traffic congestion.

[0004] With the development of urbanization and artificial intelligence, data-driven control methods play an increasingly important role in intelligent transportation systems. Therefore, many researchers focus on using adaptive methods to solve traffic light control problems. In this context, reinforcement learning methods are widely used in traffic light control. Traffic light control is actually a sequential decision problem, which can be modeled as a Markov decision process and solved by reinforcement learning algorithms. The purpose of traffic light control is to minimize the waiting time of vehicles in the entire traffic network by adjusting the signal lights at intersections. In the control of single-intersection signals, Genders et al. were the first to apply deep reinforcement learning methods to traffic light control and used convolutional neural networks to approximate the Q-value function. However, in real life, traffic networks are interconnected, and the control of a certain intersection signal will inevitably affect the traffic conditions of its upstream and downstream intersections, thereby triggering a chain reaction at surrounding intersections. Therefore, many studies have explored the application of multi-agent reinforcement learning algorithms in the coordinated control of multi-intersection traffic lights, with the goal of finding a balance between centralized and decentralized models to reduce traffic congestion and minimize the waiting time of vehicles on the entire road network. For example, Van der Pool et al. proposed a multi-intersection traffic light control mechanism based on deep reinforcement learning, defined a new reward function, and adopted a migration planning technique for a smaller intersection set, linking the learning results with the maximum plus coordination algorithm for a larger intersection set, and achieved a more efficient coordinated control between multiple intersections. For another example, Casas applied the Deep deterministic policy gradient (DDPG) algorithm to the traffic light control system. DDPG uses the policy gradient method. Compared with the value update method, such as Deep Q-Network (DQN), DDPG directly optimizes the agent strategy and uses a deterministic strategy to directly give a certain action value, reducing the variance of action selection and accelerating algorithm optimization. For another example, Tian, ​​Tan, Feng et al. optimized the global learning model based on the idea of ​​self-region division, proposed the Coder algorithm, and performed a better aggregation of the output of the self-region, optimizing the coordination effect between different intersections. In order to further explore the spatiotemporal dependence of traffic data between different intersections, many scholars have proposed traffic light control based on graph reinforcement learning. Nishi et al. proposed a reinforcement learning algorithm based on graph convolutional neural network, which modeled multiple traffic intersections as a network graph. For example, Devailly et al. proposed an inductive graph reinforcement learning algorithm that combined graph convolutional network and independent DQN.For example, Wang et al. proposed a spatiotemporal multi-agent reinforcement learning framework, which uses graph neural networks to extract spatial structure information and recurrent neural networks to model system dynamics. Finally, the coordinated control of traffic lights at multiple intersections is achieved through a distributed DQN and attention mechanism.

[0005] Although the above works have dealt with the control problem of traffic lights at multiple intersections through collaborative reinforcement learning, the impact of coupled agents on the global road network has not been explicitly considered. Therefore, modeling the relationship between multiple agents and extracting the relationship between input traffic flow data through an interpretable network is of great research significance. Summary of the invention

[0006] The present invention provides a method and device for collaboratively controlling traffic lights at multiple intersections, and a readable storage medium, aiming to solve at least one of the technical problems existing in the prior art.

[0007] The technical solution of the present invention relates to a method for coordinated control of traffic lights at multiple intersections in a dynamic scene, wherein each lane at each traffic intersection is provided with a road sensor for transmitting vehicle information to a traffic control center; the method according to the present invention comprises the following steps:

[0008] S100, acquiring traffic flow information through the road sensor and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent;

[0009] S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic road networks, and to find the weighted aggregated traffic flow latent variables;

[0010] S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework;

[0011] S400, using the method of centralized training and distributed execution, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200;

[0012] S500. According to the strategy function and joint value function obtained through training, the optimal solution of the multi-agent reinforcement learning algorithm is obtained for application in the coordinated control of traffic lights at multiple intersections.

[0013] Further, in step S100,

[0014] The traffic flow information includes the number of vehicles n on lane j j , the density of vehicles on lane j The waiting time W at intersection i i (k), at time k, they are expressed as follows:

[0015]

[0016] In the formula, u j ,d j denote the upstream node and downstream node of lane j respectively, represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, Represents the downstream node d j adjacent intersections, τ represents the interval coefficient between vehicles, L(e j ) represents the lane length of lane j, Indicates that the upstream node is u j and the downstream node is v i The waiting time of vehicles in the lane.

[0017] Furthermore, in step S100, based on the partially observable Markov decision mechanism, at time step k, each agent i is used to represent the joint state of the agents S = o1×o2×…o N Local observation o i It is expressed as follows:

[0018]

[0019] In the formula, the observation value of each agent i is the traffic intersection v i The length of the vehicle queue on the adjacent road ρ i =a i is the traffic light phase of the current intersection used to represent the agent's action, which is a four-bit one-hot encoding that effectively encodes the current green light phase state; Indicates the road density of the current intersection; wherein each traffic intersection is provided with a separate controller for providing a signal light phase;

[0020] Among them, the reward function of the agent is expressed as follows:

[0021]

[0022] Where κ1, κ2, κ3 represent the hyperparameters of the reward function under different traffic intersections, ΔQ i(k) represents the change of vehicle queues between adjacent time steps;

[0023] The global reward on the entire traffic network is set to the reward r for each agent. i The linear sum r(k) is expressed as follows:

[0024]

[0025] In the formula, N represents the number of agents.

[0026] Further, in step S200,

[0027] All hidden layer nodes of the efficient link neural network are connected to the output layer, and the neurons in the hidden layer of the efficient link neural network include source nodes D and intermediate nodes C;

[0028] Among them, the importance coefficients of input data on different traffic networks are expressed as follows:

[0029]

[0030] In the formula, the variance σ m As input data for different traffic routes The importance coefficient of VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where f(·) is the output function of the efficient link neural network; represents the input of the efficient link neural network at time step k obtained according to the traffic flow information; α p,s is the weight coefficient of the efficient link neural network, z p,s is the output of the intermediate node C;

[0031] Among them, the traffic flow latent variable after weighted aggregation is expressed as follows:

[0032]

[0033] In the formula, H i,k is the latent variable output by the efficient link neural network at the i-th intersection at time step k.

[0034] Further, in step S300,

[0035] The neuron z(x) of the biased ReLU neural network is expressed as follows:

[0036]

[0037] In the formula, q iRepresents the number of segmentation biases on the i-th dimension of the input data. The biased ReLU neural network uses different bias parameters in different dimensions.

[0038] Further, in step S300,

[0039] The optimal solution of reinforcement learning algorithm π * It is expressed as follows:

[0040]

[0041] u∈π(x)

[0042] Where x represents the state in reinforcement learning, w represents a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor; is the optimal value function, and f(x,u,w) represents the state transition function of reinforcement learning.

[0043] Further, in step S400,

[0044] By summing up the loss functions of all critics, we get a joint value function L(φ). The goal of the critic is to minimize the joint value function, which is expressed as follows:

[0045]

[0046] In the formula, represents the advantage function, N b represents the training batch size, b ′ is any sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficients of the neural network;

[0047] Each decentralized actor updates its policy function π based on the state variables observed locally in the traffic network. i,θ , and use the clip function to limit the change ratio of the new and old strategies L i (θ):

[0048]

[0049] In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations.

[0050] Furthermore, in step S500, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network, which is expressed as follows:

[0051]

[0052] Among them, T s Indicates the test time, E i (t) represents agent i at test time T s The average waiting time of vehicles in the area.

[0053] The technical solution of the present invention also relates to a computer-readable storage medium on which program instructions are stored, and the above-mentioned method is implemented when the program instructions are executed by a processor.

[0054] The technical solution of the present invention also relates to a coordinated control device for traffic lights at multiple intersections, wherein the device includes a computer device, and the computer device includes the above-mentioned computer-readable storage medium.

[0055] The beneficial effects of the present invention are as follows:

[0056] The present invention provides an interpretable influence mechanism to reveal the spatiotemporal dependency of input traffic flow data, and proposes a multi-agent reinforcement learning algorithm based on the influence mechanism. In the scenario of multi-intersection traffic lights, the multi-agent reinforcement learning algorithm based on the interpretable influence mechanism proposed by the present invention is used. On the one hand, the spatiotemporal correlation of traffic characteristics is extracted through the interpretable influence mechanism, and the influence of different input traffic data on the output is modeled. On the other hand, the multi-agent reinforcement learning algorithm is used to realize the coordinated control of traffic lights at multiple intersections, alleviate intersection congestion, and effectively improve the efficiency and safety of road traffic flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a basic flow chart of the method according to the present invention.

[0058] Figure 2 It is a schematic diagram of the structure of a highly efficient linked neural network according to the method of the present invention.

[0059] Figure 3 Schematic diagram of the structure of the biased ReLU neural network according to the method of the present invention.

[0060] Figure 4 Flowchart of a multi-agent reinforcement learning algorithm according to the method of the present invention.

[0061] Figure 5 It is a structural schematic diagram of a simulated irregular traffic network according to the method of the present invention.

[0062] Figure 6 The figure is a comparison result of the intersection waiting time between the method according to the present invention and various existing methods.

[0063] Figure 7This is a comparison chart of the average reward per round between the method according to the present invention and the existing advanced algorithms.

[0064] Figure 8 It is a comparison chart of the stability of road network traffic flow in each round according to the method of the present invention and the existing advanced algorithm. DETAILED DESCRIPTION

[0065] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0066] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0067] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0068] Reference Figures 1 to 4 In some embodiments, the method for coordinated control of traffic lights at multiple intersections according to the present invention comprises at least the following steps:

[0069] S100, obtaining traffic flow information through road sensors and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent;

[0070] S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic networks, and to find the weighted aggregated traffic flow latent variables;

[0071] S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework;

[0072] S400, using the method of centralized training and distributed execution, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200;

[0073] S500. According to the strategy function and joint value function obtained through training, the optimal solution of the multi-agent reinforcement learning algorithm is obtained for application in the coordinated control of traffic lights at multiple intersections.

[0074] The present invention proposes a method for collaborative control of traffic lights at multiple intersections based on an explainable influence mechanism, which is a method for collaborative control of large-scale traffic lights in an irregular road network. It can better establish the relationship between different intersections and realize collaborative control of traffic lights in the global traffic network.

[0075] The method of the present invention can coordinately control the entire traffic light including multiple traffic networks, effectively alleviate the congestion at the intersection, and improve the efficiency and safety of the road traffic flow. The traffic network can be an irregular traffic network, a multi-multi regular traffic network, etc. Each traffic network can have multiple traffic intersections, and the traffic intersections can be three-way intersections and / or crossroads, etc. Each traffic intersection can be provided with multiple two-way lanes and / or one-way lanes in different directions. Among them, the lane length between two adjacent traffic intersections, the speed limit of each lane, and the number of external input lanes can be obtained through the traffic control center. Furthermore, the traffic light time of the traffic intersection can be obtained through the traffic control center, including the yellow light time and the minimum and maximum green light time.

[0076] Specific implementation of step S100

[0077] In the method of the present invention, when a vehicle enters the coverage area of ​​the intelligent transportation system, the road sensor collects statistics on the vehicle information and transmits the information to the traffic control center, defines the variables of the state, action, and reward function in the reinforcement learning, and stores them in the memory bank.

[0078] Specifically, the traffic flow information on each traffic network at time k is obtained through the road sensors on each lane on the traffic network. The present invention models the traffic network as a graph network, where each intersection is a node in the graph network and the connecting section of adjacent intersections is an edge in the graph network. The number of vehicles on lane j, n j , the density of vehicles on lane j The waiting time W at intersection i i (k), which are respectively represented as follows:

[0079]

[0080] In the formula, u j ,d j denote the upstream node and downstream node of lane j respectively, represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, Represents the downstream node d j adjacent intersections, τ represents the interval coefficient between vehicles. L(e j ) represents the lane length of lane j, that is, the lane length between two adjacent traffic intersections. Indicates that the upstream node is u j and the downstream node is v i The waiting time of vehicles in the lane.

[0081] In the traffic light control model based on the multi-agent reinforcement learning algorithm of the present invention, the multi-intersection traffic light control problem is defined as a partially observable Markov decision problem, and the state, action and reward function of each agent are given.

[0082] In some application embodiments, in order to better represent the in-and-out traffic flow on the road and the number of vehicle queues at the intersection, the present invention defines the observation value of each agent i as the traffic intersection v i (i.e., the downstream node v of lane j i )'s adjacent road queue length The signal light phase ρ and road density at the current intersection At time step k, each agent i’s local observation o i (k) can be defined as:

[0083]

[0084] Among them, i (k) is used to represent the joint state of agent i S = o1×o2×…o N Each traffic intersection has a separate controller to provide the phase of the traffic light. The signal light phase ρ is used to represent the agent action, which is a four-bit one-hot encoding that effectively encodes the current green light phase state. At each time step k, the traffic intersection v i The controller selects a discrete action, which is expressed as ρ i =a iThe number of joint actions increases exponentially with the number of agents. Among them, the present invention only considers feasible phase configurations in the action set and uses four-stage green phase control.

[0085] In some application embodiments, the present invention proposes a new reward function. Considering the characteristics of large-scale traffic networks and the optimization goal of traffic light control, the reward function of the agent is defined as:

[0086]

[0087] Where κ1, κ2, κ3 are the hyperparameters of the reward function under different traffic intersection conditions, ΔQ i (k) is the change of vehicle queues between adjacent time steps.

[0088] The global reward on the entire traffic network is set to the reward r for each agent. i The linear sum r(k) is expressed as follows:

[0089]

[0090] Where N is the number of agents, that is, the number of traffic intersections.

[0091] Specific implementation of step S200

[0092] The traffic light control model is a nonlinear system, which is very challenging to process complex spatiotemporal traffic flow data. Therefore, the present invention proposes an interpretable influence mechanism based on an efficient link neural network (EHHNN) to extract the spatiotemporal dependency of traffic flow data to enhance the coordinated control of traffic lights at different traffic intersections. Based on the interpretable influence mechanism, the present invention calculates the importance coefficient of input data on different road networks based on the road network traffic flow data of each control interval in the memory bank, and calculates the weighted aggregated traffic flow hidden variables.

[0093] First, the present invention inputs the traffic flow data obtained from step S100 into a linear transformation layer as the input of the efficient link neural network in step k. Among them, EHHNN can be regarded as a directed acyclic graph, and its network structure can be found in Figure 2 As shown, all hidden layer nodes in EHHNN are connected to the output layer, and the hidden layer includes two types of neurons, including source nodes D and intermediate nodes C.

[0094] In the Efficient Link Neural Network, the output of the source node D can be described as:

[0095]

[0096] Among them, m represents the dimension of the input data, Represents the input variable x m Previous q m A bias parameter.

[0097] In the hidden layer of EHHNN, the output z of the intermediate node C is p,s Obtained by taking the minimum value of the neurons generated in the first few layers:

[0098]

[0099] in, is defined as including all neurons generated by the previous layer of the network, |J p,s |=p represents the number of intermediate nodes in the p-th layer network.

[0100] Finally, the output of the EHHNN is the weighted sum of all neurons in the hidden layer:

[0101]

[0102] where α 1,s ,α p,s are all weight coefficients of EHHNN, α0 is the bias constant, n1,n p Represent the number of neurons in the 1st and pth layers of the network respectively.

[0103] In this paper, a two-way analysis of variance (ANOVA) is used to determine the main effects of different traffic flow information as individual factors, as well as the interactive effects of bivariate factors on intersection congestion. This feature of EHHNN provides insights into how different variables contribute to the overall prediction and helps to gain a deeper understanding of the underlying relationships in the data, so that hidden nodes that affect each output component in the ANOVA can be identified:

[0104]

[0105] In the formula, VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where the variance σ m As input data for different traffic routes The importance coefficient of , where f(· ) is the output function of the efficient link neural network, σ m The larger the value of The greater the impact on the prediction output.

[0106] The present invention realizes accurate prediction based on the influence mechanism of EHHNN on the one hand, and obtains the influence coefficient of different inputs on the output through ANOVA decomposition on the other hand, thereby deriving the weighted aggregate embedding of traffic flow information from other lanes of the traffic network, that is, the weighted aggregated traffic flow latent variable is expressed as follows:

[0107]

[0108] Among them, H i,k is the hidden variable obtained by EHHNN at the i-th intersection in the k-th step.

[0109] Specific implementation of step S300

[0110] The Actor-Critic algorithm is a strategy learning method in reinforcement learning. This paper designs a new piecewise linear neural network, namely a biased ReLU neural network, to approximate the joint value function in the actor-critic method. and the policy function π i,θ .

[0111] See also Figure 3 The network structure of the biased ReLU neural network, whose input data First it passes through the source node D (source neurons), then through the intermediate node C (intermediate neurons) before outputting The biased ReLU neural network of the present invention is a piecewise linear neural network similar to the ReLU neural network. Unlike the ReLU neural network, the biased ReLU uses different bias parameters in different dimensions.

[0112] Specifically, the bias parameter set in each dimension is defined as It is determined based on the distribution of the input data:

[0113]

[0114] Where v and η are the variance and expectation of the input data, respectively.

[0115] The neuron z(x) of the biased ReLU neural network can be expressed as:

[0116]

[0117] In the formula, q i Indicates the number of split offsets on the i-th dimension of the input data.

[0118] For a reinforcement learning problem, the optimal solution π * It can be expressed as:

[0119]

[0120] u∈π(x)

[0121] Where x represents the state in reinforcement learning, w is a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor. is the optimal value function, and f(x,u,w) represents the state transfer function in reinforcement learning.

[0122] According to the conclusions in the prior art, maximizing or minimizing a piecewise linear value function will result in a piecewise linear policy solution. The present invention constructs a piecewise linear actor-critic framework by using biased ReLU to approximate the policy function and value function in the actor-critic.

[0123] Specific implementation of step S400

[0124] See also Figure 4 Flow chart of the multi-agent reinforcement learning algorithm of the present invention. Different from the independent proximal policy optimization method (IPPO) of the prior art, the multi-agent actor-critic algorithm proposed in the present invention is a method using centralized training and distributed execution. Each agent actor is locally observed (step S100). i (k)) to obtain the traffic flow information, and train the respective strategy functions, while the centralized critic is trained to obtain a joint value function based on the weighted aggregated traffic flow latent variables obtained in step S200.

[0125] Specifically, the present invention updates all critics together to minimize the joint value function, that is, the goal of the critic is to minimize the joint value function. represents the advantage function, which is a truncated estimate of the generalized advantage estimate of the prior art, N b represents the training batch size, b ′ is any sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficient of the neural network is then obtained by summing the loss functions of all critics to obtain a joint value function L(φ), which is expressed as follows:

[0126]

[0127] Among them, L(φ) is the function to minimize the joint value, is the weighted aggregated traffic flow hidden variable obtained in step S200. Each decentralized actor updates its strategy function π according to the state variables observed locally in the traffic network. i,θ , and use the proximal strategy optimization method, and use the clip function to limit the change ratio of the new and old strategies to L i (θ):

[0128]

[0129] In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations.

[0130] Specific implementation of step S500

[0131] Repeat steps S100 to S400, and replace the strategy obtained in each time step k (the optimal solution of reinforcement learning in step S300) with * ) is applied in traffic light control until each round of coordinated control of multi-intersection traffic lights is completed. It should be noted that a round specifically refers to a complete sequence starting from the initial state of the traffic network, that is, time step k=0, until the set end state is reached. The end state of the present invention is that the simulation network time reaches 2500s.

[0132] Specifically, by controlling the signal light phase at each traffic intersection and collecting traffic flow information at each time step k through sensors on the traffic lanes, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network:

[0133]

[0134] Among them, T s Indicates the test time, E i (t) represents agent i at test time T s Average waiting time of vehicles in Figure 2 Neuron E).

[0135] A specific embodiment is used here to illustrate the method. The multi-intersection traffic light coordinated control method of the present invention is applied to two different traffic networks for simulation verification, wherein the simulation platform is Simulation of Urban Mobility (SUMO), and the simulation scene of the multi-intersection traffic light is designed as follows.

[0136] For example, the traffic network is set as a 5×5 traffic grid, in which each intersection has three two-way lanes in the four directions of east, south, west and north. The lane length between adjacent intersections is 100 meters, the speed limit of each lane is 13.89m / s, and 930 simulated vehicles enter the network in each round.

[0137] For example, for an irregular traffic network, it is set as an irregular traffic network with 8 traffic intersections, including 2 three-way intersections and 6 crossroads. The lane length between adjacent intersections is between 75 meters and 150 meters, the speed limit of each lane is 13.89m / s, there are 10 external input lanes, and 250 simulated vehicles are input into the network each round.

[0138] In addition, to ensure the safety of traffic intersections and the effectiveness of signal light output, the yellow light time of the intersection signal light is set to 2 seconds, and the minimum and maximum green light times are 5 seconds and 50 seconds respectively. By calling the traci interface, the traffic flow information on each traffic network at time k can be obtained through the sensors on each lane of the simulation platform.

[0139] Specifically, we use Python to control the phase of the traffic lights at each intersection on the SUMO simulation platform, and collect traffic flow information at each time step k through sensors on the traffic road. Figures 5 to 8 The present invention Figure 5 The experiment was carried out on a simulated irregular traffic network. Figure 6 The results of the intersection waiting time of the present invention comparing timing control, IPPO algorithm (independent policy proximal optimization), IDQN algorithm (independent deep Q-Network), MADDPG algorithm (multi-agent deep deterministic policy gradient), and DGN algorithm (Graph Convolutional Reinforcement Learning) are given. Figure 7 A comparison chart of the average reward per round between the present invention and the existing advanced algorithms is given. Figure 8A comparison chart of the stability of traffic flow in each round of the present invention and the existing advanced algorithms is given. The algorithm proposed by the present invention has better control performance, can achieve lower traffic network delays and smoother traffic conditions, and effectively alleviate traffic congestion at multiple intersections. At the same time, the present invention completed two ablation experiments. Compared with the BR-EHH-waiting-time-reward experiment, the new reward function proposed by the present invention can obtain better control effects; compared with the ReLU-EHH experiment, the actor-critic framework based on the biased ReLU neural network used by the present invention has better control effects.

[0140] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A method for coordinated control of traffic lights at multiple intersections, characterized in that: in, Road sensors are installed on each lane of each traffic intersection to transmit vehicle information to the traffic control center; The method comprises the following steps: S100, acquiring traffic flow information through the road sensor and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent; S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic road networks, and to find the weighted aggregated traffic flow latent variables; S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework; S400, using the method of centralized training and distributed execution, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200; S500, obtaining an optimal solution of a multi-agent reinforcement learning algorithm according to the strategy function and the joint value function obtained through training, so as to be applied to the coordinated control of traffic lights at multiple intersections; Wherein, in the step S100, The traffic flow information includes the number of vehicles n on lane j j , the density of vehicles on lane j The waiting time W at intersection i i (k), at time k, they are expressed as follows: In the formula, u j ,d j denote the upstream node and downstream node of lane j respectively, represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, Represents the downstream node d j adjacent intersections, τ represents the interval coefficient between vehicles, L(e j ) represents the lane length of lane j, Indicates that the upstream node is u j and the downstream node is v i The waiting time of vehicles in the lanes; Wherein, in step S100, based on the partially observable Markov decision mechanism, at time step k, each agent i is used to represent the joint state of the agents S = o1×o2×…o N Local observation o i It is expressed as follows: In the formula, the observation value of each agent i is the traffic intersection v i The length of the vehicle queue on the adjacent road ρ i =a i is the traffic light phase of the current intersection used to represent the agent's action; Indicates the road density of the current intersection; wherein each traffic intersection is provided with a separate controller for providing a signal light phase; Among them, the reward function of the agent is expressed as follows: Where k1, k2, k3 represent the hyperparameters of the reward function under different traffic intersections, ΔQ i (k) represents the change of vehicle queues between adjacent time steps; The global reward on the entire traffic network is set to the reward r for each agent. i The linear sum r(k) is expressed as follows: In the formula, N represents the number of agents; Wherein, in the step S200, All hidden layer nodes of the efficient link neural network are connected to the output layer, and the neurons in the hidden layer of the efficient link neural network include source nodes D and intermediate nodes C; Among them, the importance coefficients of input data on different traffic networks are expressed as follows: In the formula, the variance σ m As input data for different traffic routes The importance coefficient of VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where f(·) is the output function of the efficient link neural network; represents the input of the efficient link neural network at time step k obtained according to the traffic flow information; α p,s is the weight coefficient of the efficient link neural network, z p,s is the output of the intermediate node C; Among them, the traffic flow latent variable after weighted aggregation is expressed as follows: In the formula, H i,k is the latent variable output by the efficient link neural network at the i-th intersection at time step k.

2. The method according to claim 1, characterized in that: In the step S300, The neuron z(x) of the biased ReLU neural network is expressed as follows: In the formula, q i Represents the number of segmentation biases on the i-th dimension of the input data. The biased ReLU neural network uses different bias parameters in different dimensions.

3. The method according to claim 1, characterized in that In the step S300, The optimal solution π* of the reinforcement learning algorithm is expressed as follows: u∈π(x) Where x represents the state in reinforcement learning, w represents a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor; is the optimal value function, and f(x, u, w) represents the state transition equation of reinforcement learning.

4. The method according to claim 1, characterized in that: In the step S400, By summing up the loss functions of all critics, we get a joint value function L(φ), which is expressed as follows: In the formula, represents the advantage function, N b represents the training batch size, b′ is any sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficients of the neural network; Each decentralized actor updates its policy function π based on the state variables observed locally in the traffic network. i，θ , and use the clip function to limit the change ratio of the new and old strategies L i (θ): In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations.

5. The method according to claim 1, characterized in that In step S500, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network, which is expressed as follows: Among them, T s Indicates the test time, E i (t) represents agent i at test time T s The average waiting time of vehicles in the area. 6 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 5. 7.Multi-intersection traffic light coordinated control equipment, characterized in that: include: A computer device comprising a computer readable storage medium according to claim 6.

Images