Traffic signal control method based on multi-agent deep reinforcement learning

Abstract

The application relates to a traffic signal control method based on multi-agent deep reinforcement learning. The method comprises the following steps: setting an agent for an intersection, deploying a fitting network in the cloud or a server, predicting and obtaining a traffic flow prediction result Y according to a traffic network structure and historical traffic flow data; training each agent and the fitting network by using the traffic flow prediction result and real-time traffic information randomly obtained at each intersection, obtaining each optimized agent and an optimized fitting network; each optimized agent independently observes the traffic flow prediction result Y and the real-time traffic information thereof and transmits the result to the optimized fitting network, calculates a global reward and feeds back the global reward to each agent, each agent executes the next action according to the current observation result and the reward, and a current optimal traffic signal control strategy of a target area is formed. The application realizes multi-intersection traffic signal coordinated control and obviously improves travel efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular relates to a traffic signal control method based on multi-agent deep reinforcement learning. Background Art

[0002] As the number of motor vehicles continues to increase, exceeding the road's carrying capacity, traffic congestion has become a widespread and difficult problem to address. Traffic congestion reduces travel efficiency, causes environmental pollution and economic losses, and even affects drivers' health. Effectively alleviating traffic congestion and improving road capacity have become urgent challenges.

[0003] Expanding road infrastructure to cope with growing traffic volume requires significant time and resources, and in the long run, it fails to resolve the conflict between traffic demand and supply. However, building an intelligent transportation system to rationally manage and control traffic flows and improve the utilization of transportation facilities can effectively alleviate traffic congestion.

[0004] Real-time traffic information is an essential component of intelligent transportation systems. Only by understanding the current traffic conditions and network structure can urban traffic be effectively controlled. However, if traffic control is based solely on real-time data, the results will always lag behind actual traffic flow changes, failing to achieve effective signal control. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a traffic signal control method based on multi-agent deep reinforcement learning. This method predicts future short-term traffic flow information and combines the real-time traffic status of the intersection as the state input of the reinforcement learning model. The method learns the traffic signal control strategy through the interaction between the agent and the environment, thereby realizing intelligent control and coordinated control of traffic signals at multiple intersections.

[0006] This paper proposes a traffic signal control method based on multi-agent deep reinforcement learning. An agent is set up at each intersection in the target area, and a fitting network is deployed on the cloud or server. Each agent interacts with the traffic network environment and the fitting network in real time. The method specifically includes:

[0007] Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y;

[0008] Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network;

[0009] Each optimized agent performs independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of global rewards;

[0010] Each optimized agent obtains its own reward from the global reward according to its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

[0011] Furthermore, the specific process of preprocessing the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the traffic network of the target area based on the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X based on the historical traffic flow data of the target area.

[0012] Furthermore, the short-term traffic flow prediction network includes a graph convolutional neural network (GCN) and a long short-term memory network (LSTM). The spatial features of the historical traffic flow data information matrix X are extracted through GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through LSTM.

[0013] Beneficial effects of the present invention:

[0014] First, the present invention uses a pre-trained short-term traffic flow prediction network to accurately predict short-term future traffic flow information Y based on the historical traffic flow data and traffic network structure of the target area. Each intelligent agent independently observes and learns based on the predicted short-term future traffic flow information Y and the real-time traffic information at the current intersection. The observation and learning results are transmitted to the fitting network for learning and calculation. The fitting network then provides feedback to each intelligent agent. Through the interaction between the intelligent agent, the environment (i.e., the entire traffic network), and the fitting network, traffic signal control strategies are automatically learned and adjusted. This enables coordinated and intelligent control of multiple intersections, effectively alleviating traffic congestion, improving traffic efficiency, and increasing road throughput.

[0015] Secondly, the short-term traffic flow prediction network includes a graph convolutional network (GCN) and an LSTM network. GCN can effectively extract the spatial features of the traffic network, and the LSTM network can effectively extract the temporal features of traffic flow sequence data. The combination of GCN and LSTM networks can process large-scale traffic networks, comprehensively consider the temporal and spatial features of traffic flow data, and significantly improve the accuracy of predicting short-term future traffic flow information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of steps in an embodiment of the present invention;

[0017] Figure 2Schematic diagram of the overall process in an embodiment of the present invention;

[0018] Figure 3 Schematic diagram of the network prediction process for short-term traffic flow prediction in an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of a traffic signal control network modeling in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of traffic status and traffic light phases at a single intersection in an embodiment of the present invention;

[0021] Figure 6 Schematic diagram of the network structure of the multi-agent network and the fitting network in an embodiment of the present invention;

[0022] Figure 7 Schematic diagram of the training process of multiple agents and fitting networks in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] Glossary: ​​QMIX (Montonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning) algorithm: a monotonic value function factorisation algorithm for deep multi-agent reinforcement learning.

[0025] The embodiment of the present invention proposes a traffic signal control method based on multi-agent deep reinforcement learning, referring to Figure 1 、 2 As shown, an intelligent agent is set for each intersection in the target area, and a fitting network is deployed on the cloud or server. Each intelligent agent interacts with the traffic network environment and the fitting network in the target area in real time. The method specifically includes:

[0026] Step A: Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y;

[0027] Step B: Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network;

[0028] Step C: Each optimized agent performs independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of the global reward;

[0029] Step D: Each optimized agent obtains its own reward from the global reward based on its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

[0030] Figure 2 Schematic diagram of the overall process in an embodiment of the present invention. Figure 2 The overall process of the embodiment of the present invention is as follows: obtaining the traffic network structure of the target area, and obtaining its graph adjacency matrix A after data preprocessing; obtaining the historical traffic flow data of the target area, and obtaining the historical traffic flow data information matrix X after data preprocessing; inputting the graph adjacency matrix A and the historical traffic flow data information matrix X into the pre-trained short-term traffic flow prediction network to obtain the traffic flow prediction result Y; each intelligent agent obtains the real-time traffic information of each intersection through the traffic network of the target area, and each intelligent agent obtains the current real-time traffic information of its own intersection through real-time interaction and combines it with the traffic flow prediction result Y i Conduct independent observation and learning, transmit the observation and learning results to the fitting network for learning and calculation of global rewards, each agent forms its own reward distribution weight (i.e., different contribution values) according to the updated network parameters and obtains its own reward from the global reward, each agent according to its current observation and learning results (state and actions ) and their respective rewards Execute the next action The optimal traffic signal control strategy for the target area is formed, and then the phase and duration of the traffic lights (i.e., traffic lights) at each intersection in the target area are controlled.

[0031] As a preferred embodiment, each optimized agent obtains its own reward from the global reward according to its own contribution value, and the contribution value is the reward distribution weight formed by each optimized agent according to its own optimized network parameters.

[0032] As a preferred embodiment, the specific process of predicting the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the target area traffic network based on the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X based on the historical traffic flow data of the target area. Specifically:

[0033] S101: Obtain the traffic network structure of the target area from the map data, and use an undirected graph G to represent the traffic network structure of the target area. The nodes in the undirected graph G represent the intersections in the traffic network, and the edges between the nodes in the undirected graph G represent the roads between the intersections in the traffic network.

[0034] G=(V,E)

[0035] Where G represents the undirected graph of the traffic network structure of the target area, V represents the node set in the undirected graph, i.e., the intersection set, V=(v1,v2,…,v N ), N represents the total number of nodes in the undirected graph G, i.e., the total number of intersections in the target area traffic network, and E represents the set of edges between nodes in the undirected graph, i.e., the set of roads between intersections.

[0036] S102: According to the undirected graph G of the target area traffic network, a graph adjacency matrix A of the traffic network structure is obtained.

[0037] Specifically: graph adjacency matrix A, Represents a set of real number matrices with dimension N×N, where N represents the total number of nodes in the undirected graph G of the target area traffic network structure, i.e., the total number of intersections in the target area traffic network. The element A in the graph adjacency matrix A is i,j Represents any i-th node v i With the jth node v j The relationship between the i-th node v i With the jth node v j When connected, the i-th intersection v i and the jth intersection v j If there is a road between i,j =1, otherwise A i,j =0. A i,j =0 The corresponding position of two intersections that are not connected by roads is 0 in the adjacency matrix.

[0038] The historical traffic flow data information matrix X is specifically expressed as:

[0039] X=(x t-T+1 ,x t-T+2 ,…,x t ),

[0040] Where X represents the historical traffic flow data information matrix of the target area, Represents a set of real number matrices with dimensions of C×N×T, where C represents the traffic characteristic information dimension of the intersection, including the traffic volume and average vehicle speed of each intersection, N represents the total number of intersections in the target area traffic network, T represents the sampling time, i.e., the time step, and x t Represents the traffic flow and average vehicle speed information of all intersections in the traffic network structure of the target area at time t, x t-T+1 represents the traffic flow and average vehicle speed information of all intersections in the target area traffic network structure at time t-T+1; t-T+2 Represents the traffic flow and average vehicle speed information of all intersections in the target area traffic network structure at time t-T+2.

[0041] Specifically, the system continuously collects vehicle GPS data and roadside camera data from the target area's traffic network over a period of time. Abnormal data is then repaired or deleted, and missing data is filled in to ensure data integrity and accuracy. Traffic flow and average vehicle speed are calculated at five-minute intervals. Finally, the data is mapped to a range of 0 to 1 using a min-max normalization method to facilitate the subsequent training of the short-term traffic flow prediction network.

[0042] Figure 3 Schematic diagram of the network prediction process for short-term traffic flow prediction in an embodiment of the present invention. Figure 3 In the short-term traffic flow prediction network, a graph convolutional neural network (GCN) and a long short-term memory network (LSTM) are used. The graph adjacency matrix A of the target area's traffic network and the historical traffic flow data matrix X of the target area are input into the short-term traffic flow prediction network. The GCN extracts the spatial features of the historical traffic flow data matrix X, while the LSTM extracts the temporal features of the historical traffic flow data matrix X. The final output is the traffic flow prediction result Y.

[0043] As a preferred embodiment, refer to Figure 3 As shown, the pre-trained short-term traffic flow prediction network includes a graph convolutional neural network (GCN) and a long short-term memory (LSTM) network. The spatial features of the historical traffic flow data information matrix X are extracted through GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through LSTM. Specifically:

[0044] S201: Extract the spatial features of the historical traffic flow data information matrix X through GCN:

[0045] Because traffic networks typically exhibit irregular graph structures and represent non-Euclidean spatial data, GCN is more suitable for processing such non-Euclidean spatial data. GCN aggregates information between adjacent intersections by performing Laplace and Fourier transforms on the graph adjacency matrix A of the traffic network structure, thereby capturing the impact of traffic flow at adjacent intersections on the traffic flow at the own intersection.

[0046] The transfer function between graph convolutional layers is:

[0047] H (l+1) =f(H (l) ,A)

[0048]

[0049] Among them, H (l+1) represents the input features of the l+1 layer, H (l) represents the input features of the lth layer, When l=1, N represents the total number of nodes, C represents the dimension of the node's traffic characteristic information, T represents the sampling time; A represents the graph adjacency matrix of the traffic network structure; in, represents the adjacency matrix considering its own node information, and I represents the identity matrix; Where D represents the degree matrix of A, express The degree matrix of σ(·) represents the sigmoid activation function; represents the learnable parameters and F represents the output dimension.

[0050] When two graph convolution layers are used to extract spatial features, the specific transfer method is:

[0051]

[0052] Among them, Z represents the output feature matrix after two layers of graph convolution layers, X represents the input historical traffic flow feature information matrix, and A represents the graph adjacency matrix of the traffic network structure. W (0) Represents the weight matrix of the first graph convolution layer, W (1) They represent the weight matrix of the second graph convolutional layer, and σ(·) represents the sigmoid activation function.

[0053] S202: Extracting the time features of the historical traffic flow data information matrix X through the LSTM network.

[0054] Traffic flow data is time-series data. We build an LSTM network to parse this data and extract temporal features. An LSTM network is a special type of recurrent neural network that can learn long-term dependencies in input sequences and address the problems of vanishing and exploding gradients. An LSTM network is typically composed of multiple LSTM units. Each LSTM unit consists of an input gate i, a forget gate f, an output gate o, and a memory unit c.

[0055] The input x at the current time step t and the hidden state h of the previous time step t-1 Input LSTM unit. The input gate can control the current input information to update the state of the LSTM unit. Its calculation method is i t =σ(W i ·[h t-1 ,x t ]+b i ), W i is the input gate weight parameter, b i is the input gate bias parameter.

[0056] The forget gate is responsible for controlling the memory of the LSTM unit and deciding which information to forget. Its calculation method is: t =σ(W f ·[h t-1 ,x t ]+b f ), where W f is the forget gate weight parameter, b f is the bias parameter of the forget gate.

[0057] The output gate controls the output of the LSTM unit at the current moment, thereby affecting the hidden state of the LSTM unit. Its calculation method is o t =σ(W o ·[h t-1 ,x t ]+b o ), W o is the output gate weight parameter, b o is the output gate bias parameter.

[0058] The candidate memory is calculated as is the weight parameter, is the bias parameter.

[0059] The input gate controls how much of the candidate memory is used, and the forget gate controls how much of the past memory is retained. The calculation method is: c t-1 For the memory of the past, Candidate memory.

[0060] Finally, the hidden state is controlled by the output gate, which is calculated as h t = o t · tanh(c t ).

[0061] During the whole process, sigma and tanh are activation functions, and the traffic flow features that have a greater impact on the current traffic state are retained through the gate structure, and the traffic flow features that have a smaller impact are ignored.

[0062] The traffic flow prediction result Y includes the vehicle flow and the average vehicle speed of all intersections in the target area traffic road network in the future H time steps.

[0063] Specifically, the traffic flow data of the target area traffic road network in the past T time steps is input into the trained short-term traffic flow prediction network to obtain the traffic flow information Y in the future H time steps, which is specifically represented as:

[0064]

[0065] Where y t+H represents the vehicle flow and the average vehicle speed information of all intersection nodes in the target area in the future H time steps, N represents the total number of intersections in the target area traffic road network, and C represents the traffic feature information dimension of the intersection, which specifically includes the vehicle flow and the average vehicle speed of each intersection.

[0066] In some embodiments, the short-term traffic flow prediction network uses mean square error as a loss function during pre-training, and the specific calculation formula is:

[0067]

[0068] Where LOSS represents the loss function of the short-term traffic flow prediction network, N represents the total number of intersections in the target area traffic road network, y i represents the true value of the traffic flow of the i-th intersection, represents the predicted value of the traffic flow of the i-th intersection.

[0069] Each agent obtains real-time traffic information by interacting with the target area traffic road network environment, which specifically includes obtaining real-time vehicle position matrix and vehicle speed information matrix of the current respective intersection.

[0070] Figure 4 A modeling schematic diagram of the traffic signal control network in the embodiment of the application. Figure 4In this paper, an agent is deployed at each intersection in the target area's traffic network (i.e., a node in the undirected graph G of the target area's traffic network structure). Each agent can make independent decisions based on local information. At each time step, the agent receives the traffic flow prediction result Y and real-time traffic information output by the short-term traffic flow prediction network, conducts local observations, and selects a local action based on the traffic signal control strategy (i.e., controlling the phase and duration of the traffic light at a single intersection). The local actions form a global action that is fed back to the environment (i.e., the entire traffic network in the target area, including all its intersections). After receiving the global action, the environment undergoes state transition, obtaining a global reward and the state for the next time step. The traffic signal control strategy is optimized by maximizing the global reward.

[0071] Based on deep reinforcement learning, a multi-agent model is constructed for each traffic signal control problem in the target area's traffic network structure. An agent is set up at each intersection in the target area's traffic network, and a fitting network is set up on the cloud or server side. Each agent interacts with the target area's traffic network environment and the fitting network in real time. Figure 4 As shown in the figure, each agent (a part of the entire road network) can obtain the current traffic flow prediction result Y of its own intersection based on local information (i.e. the real-time traffic information of the current intersection and the traffic flow prediction result Y i ) independently make observations, learn and perform actions, transmit the observations and learning results of all agents to the fitting network, learn and feed back the learning results to each agent (each agent receives its own reward), and each agent makes a signal light control strategy for the entire traffic network. Then all the intersections in the entire target area traffic network form a multi-agent deep reinforcement learning network based on a distributed execution architecture. In the target area traffic network, at each time step, a single agent receives the traffic flow prediction result Y output by the short-term traffic flow prediction network and the traffic flow prediction result Y of the current intersection. i , perform local observation and obtain their current states action The states of all agents action The global state s that constitutes the entire traffic network environment j and global action u j ; Each agent will state Transmitted to the fitting network, the global reward r is obtained after learning and calculation j , the traffic signal control strategy is optimized by maximizing the global reward, and the best traffic signal control strategy for the entire network is obtained.

[0072] In some embodiments, in order to represent the traffic information of each lane about to enter the intersection (including the position information and speed information of each vehicle in each lane) and comprehensively describe the real-time traffic information, discrete traffic state coding is used to convert the traffic information of each lane about to enter the intersection into a form that can be learned by the intelligent agent.

[0073] Figure 5 Schematic diagram of traffic status and traffic light phases at a single intersection in an embodiment of the present invention. Figure 5 In the example, there are lanes entering an intersection from four directions. Each lane is a three-lane, two-way traffic lane. Each lane is divided into a number of discrete grids of equal length, each grid accommodating at most one vehicle. The vehicle information within the grid is converted into a vehicle position matrix and a velocity matrix. The traffic light at this intersection has four phases: east-west straight green, east-west left turn green, north-south straight green, and north-south left turn green. Each phase lasts for a certain period of time and cycles in a fixed order.

[0074] Reference Figure 5 As shown, within a 200-meter radius of the intersection, each lane entering the intersection is divided into several discrete grids of equal length. Each grid can accommodate at most one vehicle, and the vehicle information within the grid is converted into the vehicle's position matrix and velocity matrix. The setting of the discrete grid length is very important. If it is set too long, the vehicle's position information will be inaccurate, while if it is set too short, the state space will increase. Therefore, the length of the discrete grid is set to be slightly larger than the average length of an ordinary vehicle, and in this embodiment, it is set to 8 meters. If a vehicle exists within the discrete grid, the corresponding position element in the position matrix P is 1, otherwise it is 0. In the velocity matrix V, vehicles with a speed less than 0.1 m / s are considered to be in a waiting state, and the vehicle's speed is represented by 0. Otherwise, the actual vehicle speed is used.

[0075] Get the traffic flow and average vehicle speed of the next H time steps at the i-th intersection from the traffic flow prediction result Y, and use Y i To express.

[0076] In some embodiments, the roadside units in the entire traffic network environment can provide real-time feedback on traffic information at each intersection, including traffic flow, vehicle ID, location information and speed information of each vehicle, etc. After processing, the location information matrix and speed information matrix of vehicles at each intersection are obtained.

[0077] Traffic regulations require that vehicles can turn right regardless of whether the light is red or green. Therefore, traffic lights are divided into four phases: east-west straight green light, east-west left turn green light, north-south straight green light, and north-south left turn green light. Each phase lasts for a certain period of time and cycles in a fixed order.

[0078] In some embodiments, an intelligent agent is set up at each intersection in the target area traffic network, and the position matrix and speed matrix of vehicles entering the lanes of each intersection are used to describe the traffic status of each intersection. The traffic lights at each intersection are divided into four phases. The green light in each phase lasts for n time steps. After the green light duration ends, it switches to the next phase. The four phases are cycled in a fixed order.

[0079] As a preferred embodiment, each agent obtains its own real-time traffic information by interacting with the target area's traffic network environment in real time, including the current real-time vehicle position matrix P and vehicle speed matrix V at each intersection. The results of each agent's independent observation and learning include its own state and action. The state of each agent is specifically defined as:

[0080] o i =[P i ,V i ,Y i ],

[0081] Among them, i represents the state of the i-th agent, P i represents the position matrix of the vehicle at the i-th intersection, V i represents the speed matrix of each vehicle at the i-th intersection, Y i It represents the traffic volume and average vehicle speed of the i-th intersection in the next H time steps obtained from the traffic flow prediction result Y.

[0082] Define the state of each agent o i It is a local state in the entire traffic network environment. Therefore, the global state of the entire traffic network environment is the set of all agent states (local observations), which is specifically expressed as:

[0083] s=[P,V,Y],

[0084] Where s represents the global state of the traffic network environment in the target area, P represents the position matrix of vehicles entering all lanes at the intersection, V represents the speed matrix of vehicles entering all lanes at the intersection, and Y represents the traffic flow and average vehicle speed in all lanes at the intersection in the next H time steps obtained from the traffic flow prediction result Y.

[0085] The action space of each agent is specifically expressed as:

[0086] u i ={Keep the current phase of the traffic light and switch the traffic light to the next phase}

[0087] Where u i represents the action space of the i-th agent.

[0088] In some embodiments, every five time steps, the agent determines whether to switch the current signal phase based on the input state. Therefore, the action space is set to {maintain the current traffic light phase, switch the traffic light to the next phase}. When the agent chooses to maintain the current phase, the current phase duration is extended by 5 seconds. When the agent chooses to switch to the next phase, the current phase is switched to the next phase according to the phase sequence table. Before the switch, a 3-second yellow light transition is used to ensure the safe passage of vehicles in the previous phase. To prevent the agent from frequently switching phases in a short period of time, which could cause vehicles to be unable to pass through the intersection in time, a minimum phase duration of 30 seconds is set. The agent can only select an action after the current phase duration exceeds the minimum phase duration. To prevent the agent from tending to maintain the current phase for a long time, which could result in vehicles in other directions being unable to pass in time, a maximum phase duration of 90 seconds is set. When the maximum phase duration is exceeded, the agent is forced to switch to the next phase according to the phase sequence table.

[0089] The actions u of each agent i It is a local action in the entire traffic network environment. Therefore, the global action of the entire traffic network environment is the current action u of all agents. i A collection of .

[0090] The role of rewards is for the fitted network to provide feedback to the multi-agent about its performance on previous actions. It is important to define the reward function appropriately so that it can correctly guide the learning process and help it adopt the best action strategy.

[0091] In traffic signal control, most evaluation criteria such as queue length, total vehicle waiting time, and system delay are used to determine whether a control strategy has improved traffic conditions. In this embodiment of the present invention, it is desirable to have as many vehicles as possible pass through the intersection and to have vehicles travel as fast as possible within a safe range. Therefore, this embodiment of the present invention defines the reward function as follows:

[0092]

[0093] Among them, r j represents the global reward calculated by fitting the network at the jth time step, w1 represents the first weight coefficient, l i represents the traffic flow through the i-th intersection at the j-th time step, v m represents the speed of the mth vehicle in all intersections, v max represents the maximum speed allowed by the road at the i-th intersection, and w2 represents the second weight coefficient.

[0094] The QMIX reinforcement learning algorithm is applied to obtain the phase and duration of traffic lights (i.e., red and green lights) at each intersection.

[0095] The value decomposition network (VDN) was proposed by the DeepMind team in 2018. The core of this method is to decompose the global Q(s,a) value into each local Q i (s i ,a i ), each agent has its own local value function.

[0096] QMIX, a multi-agent reinforcement learning collaboration algorithm based on value function decomposition, uses a centralized training and distributed execution architecture. This allows it to leverage global information to optimize objectives during training and make independent decisions based on local observations during execution. The core idea of ​​this algorithm is that the joint action that maximizes the global value is equivalent to the joint of local actions that maximize each local value. To meet this requirement, it is necessary to ensure that the global value function is monotonically increasing with respect to each individual value function.

[0097] Because the original QMIX algorithm suffers from Q-value overestimation, the agent network is redesigned to consist of an estimation network and a target network. These networks share the same structure, consisting of multiple fully connected layers. When calculating the target Q-value, instead of directly maximizing all Q-values ​​calculated by the target network, the estimation network first selects the action corresponding to the maximum Q-value, and the target network then calculates the target Q-value based on this action, thus mitigating the overestimation problem. Furthermore, due to its insufficient ability to fit the global Q-value function, the fitting network was increased to three layers, and the absolute value method used to generate non-negative weights was replaced with a softmax method to improve fitting capabilities.

[0098] In the present invention, each agent and fitting network adopts a reinforcement learning network architecture, including an estimation network and a fitting network, forming a multi-agent reinforcement learning network. Using the traffic flow prediction results Y, the multi-agent deep reinforcement learning network is trained using the Montonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (QMIX) algorithm.

[0099] Figure 6 Schematic diagram of the structure of the network of the intelligent agent and the fitting network in an embodiment of the present invention.

[0100] Reference Figure 6 As shown, the network structure of each agent (based on reinforcement learning) remains consistent, but their parameters are different. Each agent receives the current independent observation Learn their respective value functions represents the state of the ith agent (local observation), t represents the training step or time step, Represents the action of the i-th agent. Subsequently, the value function of each agent passes through the fitting network and outputs the global value function Q tot (s t ,u t ),s t Represents the global state of the target area traffic network environment, u t Represents the global action of the target area traffic network environment (the joint action of all agents). The weight of the fitting network is determined by receiving the global state s t When constraining the weights to be non-negative, the Softmax output is used to replace the simple absolute value method.

[0101] As a preferred embodiment, the traffic flow prediction result Y and the real-time traffic information obtained at each intersection are used to train each agent and fit the network, which specifically includes:

[0102] S1: Initialize system parameters, set exploration rate ε, discount factor γ, learning rate, training round epoch, batch size batch, respectively.

[0103] S2: Initialize the estimated network parameters of each agent and copy them to the target network parameters, initialize the estimated network parameters of the fitting network and copy them to the target network parameters.

[0104] S3: Initialize the experience replay pool. The experience replay pool stores experience (s j ,u j ,r j ,s j+1 ), that is, the interaction data between each intelligent agent and the traffic network environment of the target area, which is used for subsequent random sampling of interaction data for network training, and j represents the time step.

[0105] S4: Obtain the traffic network structure and historical traffic flow data of the target area, and input the pre-trained short-term traffic flow prediction network after pre-processing to obtain the traffic flow prediction result Y. Each intelligent agent obtains its current traffic flow prediction result Y from the traffic flow prediction result Y. i ,Each intelligent agent obtains its own real-time traffic information through real-time interaction with the traffic network environment of the target area.

[0106] S5: Each agent uses its own real-time traffic information and traffic flow prediction result Y i Perform independent observation and learning prediction, and obtain the current state of each (i.e. local observation) through observation Action (i.e. local action) All agents are in their current state action Describe the global state s of the current traffic network environment respectively j , global action u j , the global state s j Transfer to the fitting network for learning and calculation of the global reward r j , forming experience (s j ,u j ,r j ,s j+1 ) and store it in the experience replay pool, where s j+1 Represents the next global state.

[0107] At each training step, each agent selects an action according to the ε-greedy strategy. The ε-greedy strategy is an action selection strategy in reinforcement learning that uses ε to balance exploration and exploitation. Randomly selecting an action represents exploration, while selecting based on experience represents exploitation. This strategy aims to optimize the learning process by randomly selecting an action with a probability of ε when experience is initially insufficient. As the training step length increases, ε gradually decreases. As experience increases, the agent increases its chances of exploiting its experience and selects actions with higher Q values, thereby optimizing the learning process.

[0108] S6: Randomly sample data from the experience replay pool (s t ,u t ,r t ,s t+1 ), t represents the training step, i represents the serial number of the agent, and each agent obtains its own state from the sampled data and actions Input their respective estimation networks together, and output their respective estimates Q after learning i Value, that is where θ i represents the estimated network parameters of the i-th agent; each agent selects its maximum estimated Action corresponding to the value and the next state received from the experience replay pool Input their respective target networks together, and the target Q output after learning i ′ value, that is where θ i ′ represents the target network parameter of the ith agent, and the ith agent selects its maximum estimate The action corresponding to the value is the optimal action:

[0109] S7: The global state s sampled from the experience pool t and the estimates of each agent The value is input into the estimation network of the fitting network, and the global estimate Q is output after learning tot Value, that is in Represents the parameters of the estimated network of the fitted network; the next global state s is sampled from the experience pool t+1 and the target Q′ of each agent i Value, input the estimated network of the fitting network, and output the global target Q′ after learning t0t Value, that is in The parameters of the target network representing the fitted network.

[0110] S8: Calculate the global estimate Q tot Value and global target Q′ t0t The loss between them is back-propagated according to the loss, and the estimated network parameters θ of each agent are updated in turn. i and the estimated network parameters of the fitted network After each K training steps, each agent estimates its network parameters θ i Assign to its target network parameter θ′ i , that is, θ′ i =θ i , the fitted network will estimate the network parameters Assign values ​​to its target network parameters Right now

[0111] S9: Each agent forms its own reward distribution weight based on its updated network parameters and obtains its own reward from the global reward Each agent is based on its current state action and rewards Each performs the next action Then get the next state of each agent Then update the experience and store it in the experience pool.

[0112] S10: Sample data from the experience pool for iterative training until the global estimate Q tot Value and global target Q t ' 0t The iteration stops when the loss function converges between the values ​​or the preset number of training steps is reached.

[0113] As a preferred embodiment, during the training of each agent and the fitting network, the loss function L used is specifically expressed as:

[0114]

[0115] Among them, L represents the global estimate Q calculated by fitting network tot Value and global target Q t '0t The loss between values, r j represents the global reward calculated by fitting the network at the jth time step, γ represents the discount factor, Represents the global target Q calculated by fitting the network at the jth time step t ' 0t Value, s j+1 represents the global state of the traffic network environment at the j+1th time step, u j+1 represents the global action of the traffic network environment at the j+1th time step, represents the target network parameters of the fitted network, represents the global estimate Q calculated by fitting the network at the jth time step tot Value, s j represents the global state of the traffic network environment at the jth time step, u j represents the global action of the traffic network environment at the jth time step, represents the estimated network parameters of the fitted network.

[0116] The global state and global reward function are used for network training, so that the intelligent agent (i.e., each intersection) continuously interacts with the environment (i.e., the traffic network in the target area) to learn effective action strategies. After training, each intelligent agent can perform actions based on local observations (i.e., the state of each intelligent agent), and choose to switch traffic light phases or maintain the current phase, thereby achieving global optimization and coordinated control, alleviating traffic congestion and improving travel efficiency.

[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0118] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A traffic signal control method based on multi-agent deep reinforcement learning, characterized by: An intelligent agent is set up for each intersection in the target area, and a fitting network is deployed on the cloud or server. Each intelligent agent interacts with the traffic network environment and the fitting network in the target area in real time. The method specifically includes: Step A: Obtain the traffic network structure and historical traffic flow data of the target area, and after preprocessing, input them into the pre-trained short-term traffic flow prediction network for prediction to obtain the traffic flow prediction result Y; Step B: Using the traffic flow prediction result Y and the randomly obtained real-time traffic information of each intersection, each intelligent agent and fitting network are trained to obtain each optimized intelligent agent and optimized fitting network. The training process includes: Initialize the estimated network parameters and target network parameters of each agent, initialize the estimated network parameters and target network parameters of the fitting network, and initialize the experience replay pool; Each agent obtains its current traffic flow prediction result Y from the traffic flow prediction result Y i ,i represents the sequence number of the intersection or agent, and each agent randomly obtains its own real-time traffic information; Each agent is based on its own real-time traffic information and traffic flow prediction results Y i Perform independent observations to obtain their respective states and actions j represents the sequence number of the time step, i represents the sequence number of the agent or intersection, and the fitting network calculates the global reward r based on the state of all agents j , get random experience (s j ,u j ,r j ,s j+1 ) and store it in the experience replay pool, s j Indicates the current global state, s j is the state currently observed by all agents Composed of u j Indicates the current global action, u j Actions currently observed by all agents Composed of r j represents the current global reward, s j+1 represents the global action at the next time step; Randomly sample data from the experience replay pool (s t ,u t ,r t ,s t+1 ), where t represents the number of training steps, and each agent obtains its current state from the sampled data and the next state After learning through their respective estimation networks and target networks, they output their respective estimates Q i Value and target Q' i value; The current global state s sampled from the experience pool t , each agent's estimated Q i Value and target Q' i The values ​​are input into the fitting network together, and the global estimate Q is output after the estimation network and the target network of the fitting network learn. tot value and global target Q' t0t value; Calculate the global estimate Q tot Value and global target Q' t0t The loss between the values ​​is used to back-propagate the estimated network parameters θ of each agent. i , target network parameters θ' i and the estimated network parameters of the fitted network Target network parameters Each agent forms its own reward distribution weight based on its updated network parameters and obtains its own reward from the global reward Each agent is based on its current state action and rewards Each performs the next action Then update the experience and store it in the experience pool; Sampling data from the experience pool for iterative training until the global estimate Q tot Value and global target Q' t0t The iteration stops when the loss function between the values ​​converges or the preset number of training steps is reached; Step C: Each optimized agent performs independent observations based on the traffic flow prediction result Y and its current real-time traffic information, and transmits all its independent observation results to the optimized fitting network for learning and calculation of the global reward; Step D: Each optimized agent obtains its own reward from the global reward based on its own contribution value, and performs the next action based on its current observation results and its own reward to form the current optimal traffic signal control strategy for the target area.

2. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Each agent obtains its own real-time traffic information by interacting with the target area's traffic network environment in real time, including the current real-time vehicle position matrix P and vehicle speed matrix V at each intersection. The results of each agent's independent observation include its own state and action. The state of each agent is specifically expressed as: o i =[P i ,V i ,Y i ], Among them, i represents the state of the i-th agent, P i represents the position matrix of the vehicle at the i-th intersection, V i represents the speed matrix of each vehicle at the i-th intersection, Y i represents the traffic volume and average vehicle speed of the i-th intersection in the next H time steps obtained from the traffic flow prediction result Y; The action space of each agent is defined as follows: u i ={Keep the current phase of the traffic light and switch the traffic light to the next phase} Where u i represents the local action space of the i-th agent.

3. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: During the training of each agent and fitting network, the loss function L used is: Among them, L represents the global estimate Q calculated by fitting network tot Value and global target Q' t0t The loss between values, r j represents the global reward calculated by fitting the network at the jth time step, γ represents the discount factor, Represents the global target Q' calculated by fitting the network at the jth time step t0t Value, s j+1 represents the global state of the traffic network environment at the j+1th time step, u j+1 represents the global action of the traffic network environment at the j+1th time step, represents the target network parameters of the fitted network, represents the global estimate Q calculated by fitting the network at the jth time step tot Value, s j represents the global state of the traffic network environment at the jth time step, u j represents the global action of the traffic network environment at the jth time step, represents the estimated network parameters of the fitted network.

4. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Each optimized agent obtains its own reward from the global reward according to its own contribution value, which is the reward distribution weight formed by each optimized agent according to its own optimized network parameters.

5. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The specific process of preprocessing the traffic network structure and historical traffic flow data of the target area includes: obtaining the graph adjacency matrix A of the traffic network of the target area based on the traffic network structure of the target area; and obtaining the historical traffic flow data information matrix X based on the historical traffic flow data of the target area.

6. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The short-term traffic flow prediction network includes a graph convolutional neural network (GCN) and a long short-term memory (LSTM) network. The spatial features of the historical traffic flow data information matrix X are extracted through the GCN, and the temporal features of the historical traffic flow data information matrix X are extracted through the LSTM.

7. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The short-term traffic flow prediction network uses mean square error as the loss function during the pre-training process, and its specific calculation formula is: Among them, LOSS represents the loss function of the short-term traffic flow prediction network, N represents the total number of intersections in the target area traffic network, and y i represents the true value of the traffic flow at the i-th intersection, It represents the predicted value of traffic flow at the i-th intersection, where i represents the sequence number of the intersection.

8. The traffic signal control method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The traffic flow prediction result Y includes the traffic volume and average vehicle speed of all intersections in the target area traffic network in the next H time steps.

Images