Traffic signal control method based on multi-agent deep reinforcement learning

Abstract

The invention relates to a traffic signal control method based on multi-agent deep reinforcement learning. The method comprises the following steps: setting an intelligent agent for an intersection, deploying a fitting network at a cloud end or a server, and performing prediction according to a traffic road network structure and historical traffic flow data to obtain a traffic flow prediction result Y; training each intelligent agent and a fitting network by using the traffic flow prediction result and randomly acquired real-time traffic information of each intersection to obtain each optimized intelligent agent and an optimized fitting network; each optimized intelligent agent performs independent observation according to a traffic flow prediction result Y and respective current real-time traffic information, transmits the result to the optimized fitting network, calculates a global reward and feeds back the global reward to each intelligent agent, and each intelligent agent executes the next action according to the respective current observation result and reward. And forming a current optimal traffic signal control strategy of the target area. According to the invention, the traffic signal coordination control of multiple intersections is realized, and the travel efficiency is obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular to a traffic signal control method based on multi-agent deep reinforcement learning. Background Art

[0002] As the number of motor vehicles increases year by year, it exceeds the carrying capacity of roads, and traffic congestion has become a common and difficult problem to control. Traffic congestion will reduce people's travel efficiency, cause environmental pollution and economic losses, and affect the health of drivers. How to effectively alleviate traffic congestion and improve road capacity has become an urgent problem to be solved.

[0003] It takes a lot of time and resources to expand basic road facilities to cope with the growing traffic volume. In the long run, it cannot solve the contradiction between traffic demand and supply. By building an intelligent transportation system to reasonably manage and control traffic flow and improve the utilization rate of transportation facilities, traffic congestion can be effectively alleviated.

[0004] For intelligent transportation systems, real-time traffic information is an indispensable part. Only after obtaining the current traffic conditions and road network structure can urban traffic be effectively controlled. However, if regulation is only based on real-time detected data, the regulation results will always lag behind the actual traffic flow changes and fail to achieve signal control effects. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes a traffic signal control method based on multi-agent deep reinforcement learning. This method predicts future short-term traffic flow information and combines the real-time traffic status of the intersection as the state input of the reinforcement learning model. It learns the traffic signal control strategy through the interaction between the agent and the environment, thereby realizing intelligent control and coordinated control of traffic signals at multiple intersections.

[0006] The present invention proposes a traffic signal control method based on multi-agent deep reinforcement learning. An agent is set for each intersection in the target area, and a fitting network is deployed on the cloud or server. Each agent interacts with the traffic road network environment and the fitting network in the target area in real time. The method specifically includes:

[0007] Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y;

[0008] Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network;

[0009] Each optimized agent makes independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of global rewards;

[0010] Each optimized agent obtains its own reward from the global reward according to its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

[0011] Furthermore, the specific process of preprocessing the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the traffic network of the target area according to the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X according to the historical traffic flow data of the target area.

[0012] Furthermore, the short-term traffic flow prediction network includes a graph convolutional neural network GCN and a long short-term memory network LSTM. The spatial features of the historical traffic flow data information matrix X are extracted through GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through LSTM.

[0013] Beneficial effects of the present invention:

[0014] First, the present invention uses a pre-trained short-term traffic flow prediction network to accurately predict short-term future traffic flow information Y based on the historical traffic flow data and traffic network structure of the target area. Each intelligent agent independently observes and learns in combination with the predicted short-term future traffic flow information Y and the real-time traffic information of the current intersection, and transmits all its observation and learning results to the fitting network for learning and calculation. The fitting network feeds back information to each intelligent agent, and the intelligent agent interacts with the environment (i.e., the entire traffic network) and the fitting network to automatically learn and adjust the traffic signal control strategy, which can realize multi-intersection coordinated control and intelligent control, effectively alleviate traffic congestion, improve traffic efficiency, and increase road throughput.

[0015] Secondly, the short-term traffic flow prediction network includes a graph convolutional network GCN and a LSTM network. GCN can effectively extract the spatial characteristics of the traffic network, and the LSTM network can effectively extract the temporal characteristics of traffic flow sequence data. The combination of GCN and LSTM networks can process large-scale traffic road networks, comprehensively consider the temporal and spatial characteristics of traffic flow data, and significantly improve the accuracy of predicting short-term future traffic flow information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of steps of an embodiment of the present invention;

[0017] Figure 2It is a schematic diagram of the overall process in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a short-term traffic flow prediction network prediction process in an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of traffic signal control network modeling in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the traffic status and traffic light phase of a single intersection in an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of the network structure of a multi-agent network and a fitting network in an embodiment of the present invention;

[0022] Figure 7 Schematic diagram of the training process of multiple agents and fitting networks in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Glossary: ​​QMIX (Montonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning) algorithm: a monotonic value function factorisation algorithm for deep multi-agent reinforcement learning.

[0025] The embodiment of the present invention proposes a traffic signal control method based on multi-agent deep reinforcement learning, referring to Figure 1 , 2 As shown, an intelligent agent is set for each intersection in the target area, and a fitting network is deployed on the cloud or server. Each intelligent agent interacts with the traffic road network environment and the fitting network in the target area in real time. The method specifically includes:

[0026] Step A: Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y;

[0027] Step B: Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network;

[0028] Step C: Each optimized agent makes independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of global rewards;

[0029] Step D: Each optimized agent obtains its own reward from the global reward according to its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

[0030] Figure 2 Schematic diagram of the overall process in an embodiment of the present invention. Figure 2 The overall process of the embodiment of the present invention is: obtaining the traffic road network structure of the target area, and obtaining its graph adjacency matrix A after data preprocessing; obtaining the historical traffic flow data of the target area, and obtaining the historical traffic flow data information matrix X after data preprocessing; inputting the graph adjacency matrix A and the historical traffic flow data information matrix X into the pre-trained short-term traffic flow prediction network to obtain the traffic flow prediction result Y; each intelligent agent obtains the real-time traffic information of each intersection through the traffic road network of the target area, and each intelligent agent obtains the current real-time traffic information of each intersection through real-time interaction and combines it with the traffic flow prediction result Y i The intelligent agents perform independent observation and learning, transmit their observation and learning results to the fitting network for learning and calculation of global rewards. Each intelligent agent forms its own reward distribution weight (i.e., different contribution values) according to the updated network parameters and obtains its own reward from the global reward. Each intelligent agent obtains its own reward according to its current observation and learning results (state and actions ) and their respective rewards Execute the next action The optimal traffic signal control strategy for the target area is formed, thereby controlling the phase and duration of the traffic lights (i.e., traffic lights) at each intersection in the target area.

[0031] As a preferred embodiment, each optimized agent obtains its own reward from the global reward according to its own contribution value, and the contribution value is the reward distribution weight formed by each optimized agent according to its own optimized network parameters.

[0032] As a preferred embodiment, the specific process of predicting the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the traffic network of the target area according to the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X according to the historical traffic flow data of the target area. Specifically:

[0033] S101: Obtain the traffic network structure of the target area from the map data, and use an undirected graph G to represent the traffic network structure of the target area. The nodes in the undirected graph G represent the intersections in the traffic network, and the edges between the nodes in the undirected graph G represent the roads between the intersections in the traffic network.

[0034] G=(V,E)

[0035] Where G represents the undirected graph of the traffic network structure of the target area, V represents the node set in the undirected graph, i.e., the intersection set, V = (v1, v2, …, v N ), N represents the total number of nodes in the undirected graph G, i.e., the total number of intersections in the target area traffic network, and E represents the edge set between nodes in the undirected graph, i.e., the road set between each intersection

[0036] S102: According to the undirected graph G of the target area traffic network, a graph adjacency matrix A of the traffic network structure is obtained.

[0037] Specifically: graph adjacency matrix A, represents a set of real number matrices with dimension N×N, where N represents the total number of nodes in the undirected graph G of the target area traffic network structure, i.e., the total number of intersections in the target area traffic network, and the element A in the graph adjacency matrix A i,j Represents any i-th node v i With the jth node v j The relationship between the i-th node v i With the jth node v j When connected, the i-th intersection v i With the jth intersection v j If there is a road between i,j =1, otherwise A i,j = 0. i,j =0 The corresponding position of two intersections that are not connected by roads is 0 in the adjacency matrix.

[0038] The historical traffic flow data information matrix X is specifically expressed as:

[0039] X=(x t-T+1 ,x t-T+2 ,…,x t ),

[0040] Where X represents the historical traffic flow data information matrix of the target area, represents a set of real number matrices with dimensions of C×N×T, where C represents the dimension of traffic characteristic information at the intersection, including the traffic volume and average vehicle speed at each intersection, N represents the total number of intersections in the traffic road network of the target area, T represents the sampling time, i.e., the time step, and x t It represents the traffic volume and average vehicle speed information of all intersections in the traffic network structure of the target area at time t. x t-T+1 represents the traffic volume and average vehicle speed information of all intersections in the traffic network structure of the target area at time t-T+1; x t-T+2 Represents the traffic flow and average vehicle speed information of all intersections in the traffic network structure of the target area at time t-T+2.

[0041] Specifically: Continuously collect vehicle GPS data and roadside camera data of the target area traffic network over a period of time, then repair or delete abnormal data and fill in missing data to ensure data integrity and accuracy. Calculate vehicle flow and average vehicle speed at five-minute intervals. Finally, use the min-max normalization method to map the data to between 0 and 1, which is convenient for the subsequent training of the short-term traffic flow prediction network.

[0042] Figure 3 Schematic diagram of the short-term traffic flow prediction network prediction process in an embodiment of the present invention. Figure 3 In the short-term traffic flow prediction network, a graph convolutional neural network (GCN) and a long short-term memory network (LSTM) are included. The graph adjacency matrix A of the target area traffic network and the historical traffic flow data information matrix X of the target area are input into the short-term traffic flow prediction network, and the spatial features of the historical traffic flow data information matrix X are extracted by GCN, and the temporal features of the historical traffic flow data information matrix X are extracted by LSTM; finally, the traffic flow prediction result Y is output.

[0043] As a preferred embodiment, refer to Figure 3 As shown, the pre-trained short-term traffic flow prediction network includes a graph convolutional neural network GCN and a long short-term memory network LSTM. The spatial features of the historical traffic flow data information matrix X are extracted through GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through LSTM. Specifically:

[0044] S201: Extract the spatial features of the historical traffic flow data information matrix X through GCN:

[0045] Since the traffic network structure usually presents an irregular graph structure and belongs to non-Euclidean spatial data, GCN is more suitable for processing such non-Euclidean spatial data. GCN realizes information aggregation between adjacent intersections by performing Laplace transform and graph Fourier transform on the graph adjacency matrix A of the traffic network structure, thereby achieving the purpose of capturing the impact of traffic flow at adjacent intersections on traffic flow at its own intersection.

[0046] The transfer function between graph convolutional layers is:

[0047] H (l+1) =f(H (l) ,A)

[0048]

[0049] Among them, H (l+1) represents the input features of the l+1th layer, H (l) represents the input features of the lth layer, When l = 1, N represents the total number of nodes, C represents the dimension of the traffic characteristic information of the node, T represents the sampling time; A represents the graph adjacency matrix of the traffic network structure; in, represents the adjacency matrix considering its own node information, and I represents the identity matrix; Where D represents the degree matrix of A, express The degree matrix of σ(·) represents the sigmoid activation function; represents the learnable parameters and F represents the output dimension.

[0050] When two graph convolutional layers are used to extract spatial features, the specific transfer method is:

[0051]

[0052] Among them, Z represents the output feature matrix after two layers of graph convolutional layers, X represents the input historical traffic flow feature information matrix, and A represents the graph adjacency matrix of the traffic network structure. W (0) represents the weight matrix of the first graph convolutional layer, W (1) They represent the weight matrix of the second graph convolutional layer, and σ(·) represents the sigmoid activation function.

[0053] S202: Extracting the time characteristics of the historical traffic flow data information matrix X through the LSTM network.

[0054] Traffic flow data is time series data. We build an LSTM network to parse the time series data and extract time features. The LSTM network is a special recurrent neural network that can learn long-term dependencies in the input sequence and solve the problems of gradient vanishing and gradient exploding. The LSTM network is generally composed of multiple LSTM units. An LSTM unit consists of an input gate i, a forget gate f, an output gate o, and a memory unit c.

[0055] The input x at the current time step t and the hidden state h of the previous time step t-1 Input LSTM unit. The input gate can control the current input information to update the state of the LSTM unit. The calculation method is i t =σ(W i ·[h t-1 ,x t ]+b i ), W i is the input gate weight parameter, b i is the input gate bias parameter.

[0056] The forget gate is responsible for controlling the memory of the LSTM unit and deciding which information to forget. Its calculation method is: t =σ(W f ·[h t-1 ,x t ]+b f ), where W f is the forget gate weight parameter, b f is the bias parameter of the forget gate.

[0057] The output gate controls the output of the LSTM unit at the current moment, thereby affecting the hidden state of the LSTM unit. Its calculation method is o t =σ(W o ·[h t-1 ,x t ]+b o ), W o is the output gate weight parameter, b o is the output gate bias parameter.

[0058] The candidate memory is calculated as is the weight parameter, is the bias parameter.

[0059] The input gate controls how much content in the candidate memory is used, and the forget gate controls how much content in the past memory element is retained. The calculation method is: c t-1 For the memory of the past, Candidate memory.

[0060] Finally, the hidden state is controlled by the output gate, which is calculated as h t =o t ·tanh(c t ).

[0061] In the whole process, σ and tanh are activation functions, which retain the traffic flow features that have a greater impact on the current traffic state through the gate structure and ignore the traffic flow features with less impact.

[0062] The traffic flow prediction result Y includes the traffic flow and average vehicle speed of all intersections in the target area traffic network in the next H time steps.

[0063] Specifically, the traffic flow data of the target area traffic network in the past T time steps are input into the trained short-term traffic flow prediction network to obtain the traffic flow information Y in the future H time steps, which is specifically expressed as:

[0064]

[0065] Among them, y t+H It represents the traffic flow and average vehicle speed information of all intersection nodes in the target area in the future H time step, N represents the total number of intersections in the traffic road network of the target area, and C represents the traffic characteristic information dimension of the intersection, which specifically includes the traffic flow and average vehicle speed of each intersection.

[0066] In some embodiments, the short-term traffic flow prediction network uses mean square error as the loss function during the pre-training process, and its specific calculation formula is:

[0067]

[0068] Among them, LOSS represents the loss function of the short-term traffic flow prediction network, N represents the total number of intersections in the target area traffic network, and y i represents the true value of the traffic flow at the i-th intersection, Represents the predicted value of traffic flow at the i-th intersection.

[0069] Each intelligent agent obtains real-time traffic information through the traffic road network environment in the target area, which specifically includes obtaining the real-time vehicle position matrix and vehicle speed information matrix at the current intersection.

[0070] Figure 4 The figure is a schematic diagram of traffic signal control network modeling in an embodiment of the present invention. Figure 4In the method, agents are set at each intersection in the traffic network of the target area (i.e., nodes in the undirected graph G of the traffic network structure of the target area), and each agent can make independent decisions based on local information. At each time step, the agent receives the traffic flow prediction result Y and real-time traffic information output by the short-term traffic flow prediction network, makes local observations, and selects local actions based on the traffic signal control strategy (i.e., controls the phase and duration of the traffic light at a single intersection). The local actions form global actions and are fed back to the environment (i.e., the entire traffic network of the target area, including all the intersections therein). After receiving the global action, the environment performs state transfer to obtain the global reward and the state of the next time step, and optimizes the traffic signal control strategy by maximizing the global reward.

[0071] Based on deep reinforcement learning, multi-agent modeling is performed on each traffic signal control problem in the target area traffic network structure. An agent is set up at each intersection in the target area traffic network, and a fitting network is set up on the cloud or server side. Each agent interacts with the target area traffic network environment and the fitting network in real time. Figure 4 As shown in Figure 1, each agent (a part of the entire road network) can obtain the current traffic flow prediction result Y of its own intersection based on local information (i.e., the real-time traffic information of the current intersection and the traffic flow prediction result Y i ) independently make observations, learn and execute actions, transmit the observations and learning results of all agents to the fitting network, learn and feed back the learning results to each agent (each agent gets its own reward), and each agent makes a signal light control strategy for the entire traffic network. Then all the intersections in the entire target area traffic network form a multi-agent deep reinforcement learning network based on a distributed execution architecture. In the target area traffic network, at each time step, a single agent receives the traffic flow prediction result Y output by the short-term traffic flow prediction network and the traffic flow prediction result Y of the current intersection i , perform local observation and obtain their current states action The states of all agents action The global state s that constitutes the entire traffic network environment j and the global action u j ; Each agent will change its state Transmitted to the fitting network, the global reward r is obtained after learning and calculation j , the traffic signal control strategy is optimized by maximizing the global reward, and the best traffic signal control strategy for the entire network is obtained.

[0072] In some embodiments, in order to represent the traffic information of each lane that is about to enter the intersection (including the position information and speed information of each vehicle in each lane) and comprehensively describe the real-time traffic information, discrete traffic state coding is used to convert the traffic information of each lane that is about to enter the intersection into a form that can be learned by the intelligent agent.

[0073] Figure 5 Schematic diagram of traffic status and traffic light phases at a single intersection in an embodiment of the present invention. Figure 5 In the example, there are lanes entering a certain intersection from four directions. Each lane is a three-lane two-way traffic lane. Each lane is divided into several discrete grids of equal length. Each grid can accommodate at most one vehicle. The vehicle information in the grid is converted into the vehicle position matrix and speed matrix. The traffic lights at this intersection are divided into four phases: east-west straight green light, east-west left turn green light, north-south straight green light, and north-south left turn green light. Each phase lasts for a period of time and cycles in a fixed order.

[0074] Reference Figure 5 As shown, within 200 meters of the intersection, each lane entering the intersection is divided into several discrete grids of equal length, each grid can accommodate at most one vehicle, and the vehicle information in the grid is converted into the vehicle's position matrix and speed matrix. The setting of the discrete grid length is very important. If it is set too long, the vehicle's position information will be inaccurate. If it is set too short, the state space will increase. Therefore, the length of the discrete grid is set to be slightly larger than the average length of an ordinary vehicle, and in this embodiment, it is set to 8 meters. If there is a vehicle in the discrete grid, the corresponding position element of the position matrix P is 1, otherwise it is 0; in the speed matrix V, a vehicle with a speed less than 0.1m / s is regarded as a waiting state, and the speed of the vehicle is represented by 0, otherwise it is represented by the actual vehicle speed.

[0075] From the traffic flow prediction result Y, we can obtain the traffic volume and average vehicle speed of the i-th intersection in the next H time steps, and use Y i To express.

[0076] In some embodiments, the roadside units in the entire traffic network environment can provide real-time feedback on traffic information at each intersection, including vehicle flow, vehicle ID, location information and speed information of each vehicle, etc. After processing, the location information matrix and speed information matrix of vehicles at each intersection are obtained.

[0077] According to traffic rules, vehicles can turn right regardless of whether the light is red or green, so traffic lights are divided into four phases: east-west straight green light, east-west left turn green light, north-south straight green light and north-south left turn green light. Each phase lasts for a period of time and cycles in a fixed order.

[0078] In some embodiments, an intelligent agent is set at each intersection in the target area traffic network, and the position matrix and speed matrix of vehicles entering the lanes of each intersection are used to describe the traffic status of each intersection. The traffic lights at each intersection are divided into four phases. The green light in each phase lasts for n time steps. After the green light duration ends, it switches to the next phase. The four phases are cycled in a fixed order.

[0079] As a preferred embodiment, each agent obtains its own real-time traffic information by interacting with the target area traffic network environment in real time, which includes the real-time vehicle position matrix P and vehicle speed matrix V of each intersection; the results of independent observation and learning of each agent include their own state and action, and the state of each agent is specifically defined as:

[0080] o i =[P i ,V i ,Y i ],

[0081] Among them, i represents the state of the ith agent, P i represents the position matrix of the vehicle at the i-th intersection, V i represents the speed matrix of each vehicle at the i-th intersection, Y i It represents the traffic volume and average vehicle speed of the i-th intersection in the next H time steps obtained from the traffic flow prediction result Y.

[0082] Define the state of each agent i It is a local state in the entire traffic network environment. Therefore, the global state of the entire traffic network environment is the set of all agent states (local observations), which is specifically expressed as:

[0083] s=[P,V,Y],

[0084] Among them, s represents the global state of the traffic network environment in the target area, P represents the position matrix of vehicles entering all intersection lanes, V represents the speed matrix of vehicles entering all intersection lanes, and Y represents the traffic flow and average vehicle speed of all intersection lanes in the next H time steps obtained from the traffic flow prediction result Y.

[0085] The action space of each agent is specifically expressed as:

[0086] u i ={Keep the current phase of the traffic light and switch the traffic light to the next phase}

[0087] In the formula, u i represents the action space of the ith agent.

[0088] In some embodiments, at every 5 time steps, the agent determines whether to switch the current signal phase according to the input state, so the action space is set to {maintain the current phase of the traffic light, switch the traffic light to the next phase}. When the agent chooses to maintain the current phase, the current phase time is extended by 5 seconds. When the agent chooses to switch to the next phase, the current phase will be switched to the next phase according to the phase sequence table. Before switching, a 3-second yellow light time will be used to transition to ensure the safe passage of vehicles in the previous phase. In order to avoid the frequent phase switching of the agent in a short period of time, resulting in vehicles not having time to pass through the intersection and causing safety problems, the shortest phase time is set to 30 seconds. Only when the current phase duration exceeds the shortest phase time, the agent can choose an action. In order to avoid the agent's tendency to maintain the current phase for a long time, resulting in vehicles in other directions not being released in time, the longest phase time is set to 90 seconds. When the longest phase time is exceeded, it will be forced to switch to the next phase according to the phase sequence table.

[0089] The actions u of each agent i is a local action in the entire traffic network environment. Therefore, the global action of the entire traffic network environment is the current action u of all agents. i A collection of .

[0090] The role of rewards is for the fitted network to provide feedback to the multi-agent about the performance of previous actions. It is important to define the reward function appropriately so that it can correctly guide the learning process and thus help to adopt the best action strategy.

[0091] In terms of traffic signal control, most evaluation criteria such as queue length, total vehicle waiting time and system delay are used to distinguish whether the control strategy improves the traffic conditions. In the embodiment of the present invention, it is hoped that as many vehicles as possible pass through the intersection and the vehicle speed is as fast as possible within the safety range. Therefore, the embodiment of the present invention defines the reward function as follows:

[0092]

[0093] Among them, r j represents the global reward calculated by fitting the network at the jth time step, w1 represents the first weight coefficient, l i represents the traffic flow through the i-th intersection at the j-th time step, v m represents the speed of the mth vehicle in all intersections, v max represents the maximum travel speed allowed on the road at the i-th intersection, and w2 represents the second weight coefficient.

[0094] The QMIX reinforcement learning algorithm is applied to obtain the phase and duration of the traffic lights (i.e., traffic lights) at each intersection.

[0095] The value decomposition network (VDN) was proposed by the DeepMind team in 2018. The core of this method is to decompose the global Q(s,a) value into each local Q i (s i ,a i ), and each agent has its own local value function.

[0096] QMIX, a multi-agent reinforcement learning collaboration algorithm based on value function decomposition, is a centralized training distributed execution architecture, which can use global information to optimize the target in the training phase and make independent decisions based on local observations in the execution phase. The core idea of ​​the algorithm is to make the joint action with the maximum global value equivalent to the joint of local actions that maximize each local value. To meet this condition, it is necessary to ensure that the global value function is monotonically increasing with respect to the individual value functions.

[0097] Since the original QMIX algorithm has the problem of overestimation of Q value, the agent network is designed to consist of an estimation network and a target network. The estimation network and the target network have the same structure and consist of multiple fully connected layers. When calculating the target Q value, all Q values ​​calculated by the target network are no longer directly selected by maximization. Instead, the action corresponding to the maximum Q value is first selected through the estimation network, and then the target network calculates the target Q value based on this action, thereby alleviating the overestimation problem. In addition, its ability to fit the global Q value function is insufficient, so the fitting network is increased to 3 layers, and the absolute value method used when generating non-negative weights is replaced by the Softmax method to improve the fitting ability.

[0098] In the present invention, each agent and fitting network adopts a reinforcement learning network architecture, including an estimation network and a fitting network, and the whole constitutes a multi-agent reinforcement learning network. Using the traffic flow prediction result Y, the multi-agent deep reinforcement learning network is trained using a monotonic value function factorisation for deep multi-agent reinforcement learning (QMIX) algorithm.

[0099] Figure 6 Schematic diagram of the structure of the network of the intelligent agent and the fitting network in the embodiment of the present invention.

[0100] Reference Figure 6 As shown in Figure 2, the network structure of each agent (based on reinforcement learning) remains consistent, but their parameters are different. Each agent receives the current independent observation Learn their respective value functions represents the state (local observation) of the ith agent, t represents the training step or time step, represents the action of the ith agent. Then, the value function of each agent passes through the fitting network and outputs the global value function Q tot (s t ,u t ),s t Represents the global state of the traffic network environment in the target area, u t represents the global action of the target area traffic network environment (the joint action of all agents). The weight of the fitting network is determined by receiving the global state s t The fully connected network is generated by using Softmax output to replace the simple absolute value method when restricting the weights to be non-negative.

[0101] As a preferred embodiment, the traffic flow prediction result Y and the real-time traffic information obtained at each intersection are used to train each intelligent agent and the fitting network, which specifically includes:

[0102] S1: Initialize system parameters, set exploration rate ε, discount factor γ, learning rate, training round epoch, batch size batch, respectively.

[0103] S2: Initialize the estimated network parameters of each agent and copy them to the target network parameters, initialize the estimated network parameters of the fitted network and copy them to the target network parameters.

[0104] S3: Initialize the experience replay pool. The experience replay pool stores experience (s j ,u j ,r j ,s j+1 ), namely the interaction data between each intelligent agent and the traffic network environment of the target area, which is used for subsequent random sampling of interaction data for network training, and j represents the time step.

[0105] S4: Obtain the traffic network structure and historical traffic flow data of the target area, input the pre-trained short-term traffic flow prediction network for prediction after preprocessing, and obtain the traffic flow prediction result Y. Each intelligent agent obtains its current traffic flow prediction result Y from the traffic flow prediction result Y. i ,Each intelligent agent obtains its own real-time traffic information through real-time interaction with the traffic road network environment in the target area.

[0106] S5: Each intelligent agent is based on its own real-time traffic information and traffic flow prediction result Y i Perform independent observation and learning prediction, and obtain the current state of each (i.e. local observation) through observation Action (i.e. local action) All agents are in their current state action Describe the global state s of the current traffic network environment respectively j , global action u j , the global state s j Transfer to the fitting network for learning and calculation of the global reward r j , forming experience(s j ,u j ,r j ,s j+1 ) and store it in the experience replay pool, where s j+1 Represents the next global state.

[0107] At each training step, each agent selects an action according to the ε-greedy strategy. The ε-greedy strategy is an action selection strategy in reinforcement learning. It uses ε to balance exploration and utilization. Randomly selecting an action is exploration, and selecting based on experience is utilization. The purpose of this strategy is that when the experience is insufficient at the beginning, the agent randomly selects an action according to the probability ε. As the training step increases, ε gradually decays. When the experience is sufficient, the opportunity to utilize the experience is increased, and the action with a larger Q value is selected, thereby optimizing the learning process.

[0108] S6: Randomly sample data from the experience replay pool (s t ,u t ,r t ,s t+1 ), t represents the training step, i represents the serial number of the agent, and each agent obtains its own state from the sampled data and actions They are input into their respective estimation networks and output their respective estimates Q after learning. i Value, that is where θ i represents the estimated network parameters of the ith agent; each agent selects its maximum estimate Action corresponding to the value and the next state received from the experience replay pool Input into their respective target networks, and the target Q output after learning i ′ value, that is where θ i ′ represents the target network parameter of the ith agent, and the ith agent selects its maximum estimate The action corresponding to the value, i.e. the optimal action, is expressed as:

[0109] S7: The global state s sampled from the experience pool t and the estimates of each agent The value is input into the estimation network of the fitting network, and after learning, the global estimate Q is output tot Value, that is in Represents the parameters of the estimated network of the fitted network; the next global state s sampled from the experience pool t+1 and the goal Q′ of each agent i Value, input into the estimation network of the fitting network, and output the global target Q′ after learning t0t Value, that is in Represents the parameters of the target network of the fitted network.

[0110] S8: Calculate the global estimate Q tot Value and global target Q′ t0t The loss between them is then back-propagated to update the estimated network parameters θ of each agent in turn. i and the estimated network parameters of the fitted network After each K training steps, each agent estimates its network parameters θ i Assign to its target network parameter θ′ i , that is, θ′ i =θ i , the fitted network will estimate the network parameters Assign values ​​to its target network parameters Right now

[0111] S9: Each agent forms its own reward distribution weight according to its updated network parameters and obtains its own reward from the global reward Each agent is based on its current state action and rewards Each performs the next action Then get the next state of each agent Then update the experience and store it in the experience pool.

[0112] S10: Sample data from the experience pool for iterative training until the global estimate Q tot Value and global target Q t ' 0t The iteration stops when the loss function converges between the values ​​or the preset number of training steps is reached.

[0113] As a preferred embodiment, in the training process of each intelligent agent and fitting network, the loss function L used is specifically expressed as:

[0114]

[0115] Among them, L represents the global estimate Q calculated by fitting network tot Value and global target Q t '0t The loss between values, r j represents the global reward calculated by fitting the network at the jth time step, γ represents the discount factor, represents the global target Q calculated by fitting the network at the jth time step t ' 0t Value, s j+1 represents the global state of the traffic network environment at the j+1th time step, u j+1 represents the global action of the traffic network environment at the j+1th time step, represents the target network parameters of the fitted network, represents the global estimate Q calculated by fitting the network at the jth time step tot Value, s j represents the global state of the traffic network environment at the jth time step, u j represents the global action of the traffic network environment at the jth time step, represents the estimated network parameters of the fitted network.

[0116] The global state and global reward function are used to train the network, so that the intelligent agent (i.e., each intersection) continuously interacts with the environment (i.e., the traffic road network in the target area) to learn effective action strategies. After the training is completed, each intelligent agent can perform actions based on local observations (i.e., the state of each intelligent agent), and choose to switch the phase of traffic lights or maintain the current phase, thereby achieving global optimization and coordinated control, alleviating traffic congestion, and improving travel efficiency.

[0117] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0118] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A traffic signal control method based on multi-agent deep reinforcement learning, characterized in that: An intelligent agent is set for each intersection in the target area, and a fitting network is deployed on the cloud or server. Each intelligent agent interacts with the traffic road network environment and the fitting network in the target area in real time. The method specifically includes: Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y; Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network; Each optimized agent makes independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of global rewards; Each optimized agent obtains its own reward from the global reward according to its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

2. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Each agent obtains its own real-time traffic information by interacting with the target area traffic network environment in real time, which includes the real-time vehicle position matrix P and vehicle speed matrix V of each intersection; the results of independent observation by each agent include their own state and action, and the state of each agent is specifically expressed as: o i =[P i ,V i ,Y i ], Among them, i represents the state of the ith agent, P i represents the position matrix of the vehicle at the i-th intersection, V i represents the speed matrix of each vehicle at the i-th intersection, Y i It represents the traffic volume and average vehicle speed of the i-th intersection in the next H time steps obtained from the traffic flow prediction result Y; The action space of each agent is defined as follows: u i ={Keep the current phase of the traffic light and switch the traffic light to the next phase} In the formula, u i represents the local action space of the i-th agent; The definition of the global reward function is specifically expressed as: Among them, r j represents the global reward calculated by fitting the network at the jth time step, w1 represents the first weight coefficient, l i represents the traffic flow through the i-th intersection at the j-th time step, v m represents the speed of the mth vehicle in all intersections, v max represents the maximum travel speed allowed on the road at the i-th intersection, and w2 represents the second weight coefficient.

3. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 or 2 is characterized in that: The traffic flow prediction results Y and the real-time traffic information randomly obtained at each intersection are used to train each agent and fit the network, which specifically includes: Initialize the estimated network parameters and target network parameters of each agent, initialize the estimated network parameters and target network parameters of the fitting network, and initialize the experience replay pool; Each agent obtains its current traffic flow prediction result Y from the traffic flow prediction result Y i , i represents the sequence number of the intersection or agent, and each agent randomly obtains its own real-time traffic information; Each intelligent agent is based on its own real-time traffic information and traffic flow prediction results Y i Perform independent observations to obtain their respective states and actions j represents the number of the time step, i represents the number of the agent or intersection, and the fitting network calculates the global reward r based on the state of all agents. j , get random experience (s j ,u j ,r j ,s j+1 ) and store it in the experience replay pool, s j Indicates the current global state, s j is the state currently observed by all agents Composed of u j Indicates the current global action, u j Actions observed by all current agents Composed of j represents the current global reward, s j+1 represents the global action at the next time step; Randomly sample data from the experience replay pool (s t ,u t ,r t ,s t+1 ), where t represents the number of training steps, and each agent obtains its current state from the sampled data and the next state After learning through their respective estimation networks and target networks, they output their respective estimates Q i Value and target Q i ′ value; The current global state s sampled from the experience pool t , the estimated Q of each agent i Value and target Q i ′ The values ​​are input into the fitting network together, and the global estimate Q is output after the estimation network and the target network of the fitting network learn. tot value and global target Q t ′ 0t value; Calculate the global estimate Q tot Value and global target Q t ′ 0t The loss between the values ​​is used to back-propagate the estimated network parameters θ of each agent. i , target network parameters θ i ′ and the estimated network parameters of the fitted network Target network parameters Each agent forms its own reward distribution weight according to its updated network parameters and obtains its own reward from the global reward. Each agent is based on its current state action and rewards Each performs the next action Then update the experience and store it in the experience pool; Sampling data from the experience pool for iterative training until the global estimate Q tot Value and global target Q t ′ 0t The iteration stops when the loss function converges between the values ​​or the preset number of training steps is reached.

4. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: In the training process of each agent and fitting network, the loss function L used is: Among them, L represents the global estimate Q calculated by fitting network tot Value and global target Q t ′ 0t The loss between values, r j represents the global reward calculated by fitting the network at the jth time step, γ represents the discount factor, represents the global target Q calculated by fitting the network at the jth time step t ′ 0t Value, s j+1 represents the global state of the traffic network environment at the j+1th time step, u j+1 represents the global action of the traffic network environment at the j+1th time step, represents the target network parameters of the fitted network, represents the global estimate Q calculated by fitting the network at the jth time step tot Value, s j represents the global state of the traffic network environment at the jth time step, u j represents the global action of the traffic network environment at the jth time step, represents the estimated network parameters of the fitted network.

5. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Each optimized agent obtains its own reward from the global reward according to its own contribution value, and the contribution value is the reward distribution weight formed by each optimized agent according to its own optimized network parameters.

6. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 or 3, characterized in that: The specific process of preprocessing the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the traffic network of the target area according to the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X according to the historical traffic flow data of the target area.

7. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 6 is characterized in that: The short-term traffic flow prediction network includes a graph convolutional neural network (GCN) and a long short-term memory network (LSTM). The spatial features of the historical traffic flow data information matrix X are extracted through GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through LSTM.

8. The short-term traffic flow prediction network according to claim 1 uses mean square error as the loss function during the pre-training process, and its specific calculation formula is: in, LOSS represents the loss function of the short-term traffic flow prediction network, N represents the total number of intersections in the target area traffic network, and y i represents the true value of the traffic flow at the i-th intersection, It represents the predicted value of traffic flow at the i-th intersection, and i represents the sequence number of the intersection.

9. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 or 3, characterized in that: The traffic flow prediction result Y includes the traffic flow and average vehicle speed of all intersections in the target area traffic network in the next H time steps.

Images