Multi-agent reinforcement learning traffic signal coordination method considering intersection heterogeneity

Abstract

The application discloses a multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity, belongs to the field of urban traffic signal control, and particularly relates to a traffic signal cooperative control method. The application aims to solve the problem of low intersection passing efficiency. The process is as follows: data collection; simulation platform building and initialization; multi-agent deep reinforcement learning network building and initialization; the traffic state of the intersection, the traffic state of the road network and a reward function output by the simulation platform are taken as inputs of the multi-agent deep reinforcement learning network, and individual action function values of the intersections are output, so that the action feedback is determined and given to the simulation platform, and the simulation platform outputs the traffic state of the intersection, the traffic state of the road network and the reward function again after a period of time; a trained multi-agent deep reinforcement learning network is obtained; the traffic state of an actual intersection is input into the network, individual action function values are output, and the optimal action is selected and issued to the intersection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of urban traffic signal control, and particularly relates to a traffic signal cooperative control method. BACKGROUND

[0002] Traffic signal control plays an important role in improving intersection traffic efficiency, reducing traffic safety risks and reducing the travel cost of motor vehicles. According to the different control principles, the traditional traffic signal control methods can be divided into timing control, induction control and adaptive control. Among them, adaptive control can maintain the global performance level of the control system in the case of traffic flow parameters changing with time, and is the most common control method.

[0003] In recent years, with the rise of artificial intelligence, deep reinforcement learning has been widely used in traffic adaptive signal control and has achieved good control effect. This method takes the signal light as an agent, and by the powerful perception processing function of the deep learning network, the agent can autonomously observe and learn the state representation of the road network environment, and based on reinforcement learning, the agent can train the decision-making strategy to optimize the problem-solving method, realize end-to-end perception, and greatly improve the adaptive level of signal control.

[0004] However, the establishment and verification of existing deep reinforcement learning models are mostly oriented to homogeneous road networks (similar intersection levels, relatively balanced traffic flow). The differences in intersection levels, shapes, traffic flow, signal cycle length, etc. (also known as "intersection heterogeneity") often affect the action selection of researchers. However, in actual situations, in order to reasonably distribute traffic flow, the common urban road network structure is to arrange several secondary trunk roads or branch roads between two main roads, so the geometric structure and traffic flow between adjacent intersections may differ greatly, and the cycle length of the intersection may also differ greatly. The existence of heterogeneous intersections increases the uncertainty and nonlinearity of road network traffic flow, and the spatiotemporal correlation between traffic flows becomes more complex. However, existing researches are mostly oriented to homogeneous network environment and are not suitable for heterogeneous road network environment with large differences in intersection levels. SUMMARY

[0005] The purpose of the present application is to solve the problem that existing methods are mostly oriented to homogeneous network environment and are not suitable for heterogeneous road network environment with large differences in intersection levels, resulting in low intersection traffic efficiency, and to propose a multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity.

[0006] The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity specifically includes the following steps:

[0007] Step 1, data collection;

[0008] Step 2, simulation platform building and initialization;

[0009] Step 3, multi-agent deep reinforcement learning network building and initialization;

[0010] Step 4, the traffic state of the intersection, the traffic state of the road network and the reward function output by the simulation platform are taken as the input of the multi-agent deep reinforcement learning network, the individual action function value of each intersection is output by the multi-agent deep reinforcement learning network, the action is determined based on the individual action function value of each intersection output by the multi-agent deep reinforcement learning network, the action is fed back to the simulation platform through the Traci traffic control interface in the simulation platform software, and the simulation platform outputs the traffic state of the intersection, the traffic state of the road network and the reward function again after Δt seconds;

[0011] Step 5, the step 4 is repeatedly executed to train the multi-agent deep reinforcement learning network until convergence, and a trained multi-agent deep reinforcement learning network is obtained;

[0012] The trained multi-agent deep reinforcement learning network comprises:

[0013] The trained multi-agent deep reinforcement learning evaluation network and the trained multi-agent deep reinforcement learning target network;

[0014] Step 6, the traffic state of intersection i of an actual road network at a moment is input into the trained multi-agent deep reinforcement learning evaluation network, and the individual action function value of intersection i at the moment is output; The optimal action is selected based on the individual action function value, and the optimal action is issued to the intersection i.

[0015] The beneficial effects of the present application are:

[0016] The present application takes the maximization of the long-time cumulative reward of the whole road network as the target, takes the adjustment of the green light display time of the individual intersection as the action, considers the intersection heterogeneity, and constructs a value decomposition type space-time graph attention multi-agent deep reinforcement learning model for the coordinated control problem of multiple signalized intersections.

[0017] The present application takes the maximization of the long-time cumulative reward of the whole road network as the target, takes the adjustment of the green light display time of the individual intersection as the action, considers the intersection heterogeneity, and constructs a value decomposition type space-time graph attention multi-agent deep reinforcement learning model for the coordinated control problem of multiple signalized intersections.

[0018] ​The application provides a multi-agent deep reinforcement learning framework for heterogeneous intersection signal cooperative control. The framework is based on a space-time graph attention network that mines the space-time characteristics of the road network, and adjusts the signal control strategies of each intersection from the overall perspective of the road network through joint action value function decomposition, reducing the state-action space while achieving multi-agent cooperation and win-win. Secondly, in view of the actual situation that there are differences between adjacent intersections in the urban road network, a heterogeneous correlation index is designed to visualize the differences in the topological structure of the intersection, and a new reward function is designed based on this, providing a more accurate measurement method for the action selection of the agent in the heterogeneous road network environment. Finally, the introduction of the double dual network decomposes the action value function of the global road network and the individual intersection into the sum of the state value function and the action advantage function, so that the calculation of the state value function no longer depends entirely on the action, speeding up the convergence of the network, and the consistency constraint between the global optimum and the individual optimum is transformed into the value range constraint of the action advantage function, which is easy to implement, simplifying the learning process of the optimal action value function and improving the traffic efficiency of the intersection. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the application;

[0020] Figure 2 The structure diagram of the multi-agent deep reinforcement learning network, and the MLP is a multi-layer perceptron. DETAILED DESCRIPTION

[0021] Embodiment one: the specific process of the multi-agent reinforcement learning traffic signal cooperative control method considering the heterogeneity of the intersection is as follows:

[0022] Step 1, data collection;

[0023] Step 2, simulation platform building and initialization;

[0024] Step 3, multi-agent deep reinforcement learning network building and initialization;

[0025] Step 4, the traffic state of the intersection, the traffic state of the road network and the reward function output by the simulation platform are used as the input of the multi-agent deep reinforcement learning network, the individual action function value of each intersection output by the multi-agent deep reinforcement learning network is used to determine the action, and the action is fed back to the simulation platform through the Traci traffic control interface in the simulation platform software. The simulation platform outputs the traffic state of the intersection, the traffic state of the road network and the reward function again after Δt seconds;

[0026] Step 5, repeat step 4 to train the multi-agent deep reinforcement learning network until convergence, and obtain the trained multi-agent deep reinforcement learning network;

[0027] The trained multi-agent deep reinforcement learning network comprises:

[0028] The trained multi-agent deep reinforcement learning evaluation network and the trained multi-agent deep reinforcement learning target network;

[0029] Step 6, input the traffic state of the actual road network at the intersection i at the moment t into the trained multi-agent deep reinforcement learning evaluation network, and output the individual action function value of the intersection i at the moment t;

[0030] Based on the individual action function value, the optimal action is selected and issued to the intersection i.

[0031] Specific implementation method two: the difference between this implementation method and the specific implementation method one is that the data collection in step 1; the specific process is:

[0032] Step 1.1, record the intersections in the cooperative control range (each intersection in the cooperatively controlled road network) as i, and the intersection i is J i The number of entrance lanes of the intersection i (the number of entrance lanes of the intersection i is four for the cross intersection), Indicates the set of adjacent intersections of the intersection i, and I represents the number of intersections in the road network.

[0033] Record the number of entrance lanes of the intersection i as

[0034] Record the length of the entrance lane of the intersection i as The unit is m, Indicates a certain entrance lane of the intersection i.

[0035] Record the number of exit lanes of the intersection i as

[0036] Record the length of the exit lane of the intersection i as The unit is m, Indicates a certain exit lane of the intersection i.

[0037] Let P i ​​The number of signal phases of intersection i (for example, the green light starting sequence of a certain intersection is east-west straight, east-west left turn, north-south straight, and north-south left turn, so the phase number of the intersection is four, and when the green light of the north-south left turn phase ends, the green light of the east-west straight direction will start again), and the green light display time of phase p in each cycle is recorded as g i,p , p = 1, 2,..., P i ;

[0038] Step 1.2, use variable u to represent the traffic period, u = 1 represents the peak period (for example: 7:00-9:00, 17:00-19:00), and u = 0 represents the flat peak period (except the peak period);

[0039] The traffic detection equipment is used to count the traffic flow of intersection i in the peak period and the flat peak period for one week, and the data is collected as a real data set, and the hourly traffic volume of intersection i is recorded as (including peak period and flat peak period), u = 0, 1, unit: vehicle;

[0040] Step 1.3, define the number of traffic flows of phase p of intersection i as F (for a six-lane cross intersection (one lane for left straight and right straight), there are two traffic flows (east straight and west straight) in the east-west straight phase F = 2), and the key traffic flow f of phase p of intersection i is calculated by using the following formula i p ;

[0041]

[0042] In the formula, are the average hourly traffic volumes of traffic flows 1, 2, and F, respectively, unit: vehicle;

[0043] max{·} is the maximum value operation; ind{·} is the index operation (combined with the example of the above, for the east-west straight phase, the traffic volume of the east straight traffic flow is 220, and the traffic volume of the west straight traffic flow is 256, then ind{max{220, 256}} returns the traffic flow number of the west straight traffic flow);

[0044] The lane traveled by f i p is defined as

[0045] The hourly traffic volume of is recorded as unit: vehicle;

[0046] The number of lanes in the same driving direction as f i p in intersection i is recorded as

[0047] Let the number of lanes corresponding to the traffic flow of the upstream intersection i + i p be denoted as

[0048] Let the number of exit lanes of the downstream intersection i i p f - be denoted as

[0049] Let denote the distance between the intersection i and the stop line of the upstream intersection f , with the unit of m;

[0050] Let denote the distance between the intersection i and the stop line of the downstream intersection f , with the unit of m;

[0051] Step 1.4, calculate the heterogeneous correlation between the intersection i and the upstream intersection f by formula (2), denoted as

[0052]

[0053] In the formula, q represents a scaling factor for unifying the numerical range (recommended value 3);

[0054] Step 1.5, calculate the heterogeneous correlation between the intersection i and the downstream intersection f i p by formula (3), denoted as

[0055]

[0056] The other steps and parameters are the same as in the first embodiment.

[0057] Embodiment Three: The difference between this embodiment and the first or second embodiment is that the simulation platform is built and initialized in step 2. The specific process is as follows:

[0058] The simulation platform uses SUMO software;

[0059] At the beginning of the simulation, the environment time is set to 0, i.e. t = 0, t = 1, 2, …, T, T is the maximum duration of the simulation;

[0060] ​The data collected in step 1 is input into the SUMO software, and the road topology file (for the data collected in step 1.1) and the vehicle path allocation file (for the data collected in step 1.2) under different traffic periods (peak period and flat peak period) are configured in the SUMO software to realize simulation modeling of the intersection and vehicle behavior inside the intersection; the SUMO software outputs the traffic state of the intersection The traffic state S of the road network t , the reward function R t-1 .

[0061] The detector is set in the simulation road network according to the actual situation, and the maximum duration T of the simulation is set (the recommended value is 3600 seconds).

[0062] The other steps and parameters are the same as those in embodiment one or two.

[0063] Embodiment four: different from one of embodiments one to three, the data collected in step 1 is input into the SUMO software, and the road topology file (for the data collected in step 1.1) and the vehicle path allocation file (for the data collected in step 1.2) under different traffic periods (peak period and flat peak period) are configured in the SUMO software to realize simulation modeling of the intersection and vehicle behavior inside the intersection; the SUMO software outputs the traffic state of the intersection The traffic state S of the road network t , the reward function R t-1 ; the specific process is as follows:

[0064] Step 2.1, obtain the traffic state of the intersection The specific process is as follows:

[0065] Step 2.1.1, count the number of vehicles and the number of queued vehicles of the intersection i in the j direction entrance lane at time t unit: vehicle; j = 1, 2,..., J i ;

[0066] The traffic density and the queuing density of the intersection i in the j direction entrance lane at time t are calculated by formulas (4) and (5) unit: vehicle / m;

[0067] The traffic density and the queuing density of all directions of the intersection i at time t are combined to form the entrance lane traffic state of the intersection i at time t, denoted as

[0068]

[0069]

[0070] Step 2.1.2, count the number of vehicles on the exit lane of intersection i in direction j at time t Unit: vehicle;

[0071] Calculate the residual space index of the exit lane of intersection i in direction j at time t by formula (6)

[0072] Let the residual space index of all exit lanes of intersection i at time t (all lanes opposite to the J i import lanes) form the exit lane traffic state of intersection i at time t, denoted as

[0073]

[0074] In the formula, represents the maximum number of vehicles that the lane can accommodate, unit: vehicle;

[0075] Step 2.1.3, set the number of east-west straight phases to 0 and number them clockwise, count the phase number of intersection i showing green light at time t, denoted as

[0076] Let the set of represent the traffic state of intersection i at time t, denoted as

[0077] Step 2.2, obtain the traffic state S t of the road network; the specific process is:

[0078] Step 2.2.1, calculate the average traffic density of intersection i at time t by formula (7) Unit: vehicle / m;

[0079] Calculate the average queue density of intersection i at time t by formula (8) Unit: vehicle / m;

[0080] The set of both constitutes the average import lane traffic state of intersection i at time t under the road network environment

[0081]

[0082]

[0083] ​​​Step 2.2.2, calculate the average residual space indicator of intersection i at time t using formula (9)

[0084] The average exit lane traffic state of intersection i at time t is equivalent to

[0085]

[0086] Step 2.2.3, define the set of as the instantaneous traffic state of intersection i at time t in the global environment

[0087] Step 2.2.4, count the number of vehicles entering the road network in the period of Δt starting from the initial time the number of vehicles leaving the road network in units of vehicles;

[0088] Let the set represent the traffic state S of the entire road network at time t t ;

[0089] Step 2.3, obtain the reward function R t-1 ; the specific process is:

[0090] Step 2.3.1, calculate the reward that intersection i can obtain by taking action in the period of Δt starting from time t-1 using formula (10)

[0091]

[0092] where, denotes f i p the average delay per vehicle in the period of Δt starting from time t-1, in units of seconds;

[0093] denotes f i p the traffic volume in the period of Δt starting from time t-1, i.e., the number of vehicles leaving the stop line, in units of vehicles;

[0094] W i,D , W i,O are the weight values corresponding to the average delay and traffic volume, respectively, where W i,D is a negative value (recommended value -0.1), and W i,O is a positive value (recommended value 1);

[0095] Step 2.3.2, calculate the global reward R t-1The sum of the rewards of each intersection is:

[0096]

[0097] The other steps and parameters are the same as one of the first to third embodiments.

[0098] Embodiment five: the difference between this embodiment and one of the first to fourth embodiments is that the multi-agent deep reinforcement learning network is built and initialized in step 3. The specific process is as follows:

[0099] Step 3.1, build a multi-agent deep reinforcement learning network;

[0100] The multi-agent deep reinforcement learning evaluation network includes a space-time feature extraction layer (step 4.4.1), an individual value function fitting layer (step 4.4.2), and a global value function fitting layer (step 4.4.8);

[0101] The space-time feature extraction layer is based on a combination network of a fully connected neural network and GAT and GRU, which aggregates the observation values of the traffic environment of each intersection into latent representations containing adjacency intersection space-time feature information;

[0102] The individual value function fitting layer is based on a dual network to fit and calculate the state value function and action advantage value function of each intersection, and then calculate the action value function. Based on the action value function, each agent selects the optimal action based on the greedy strategy;

[0103] The global value function fitting layer takes the state value function and action advantage value function corresponding to the optimal action as input, and fits the global action function value as output;

[0104] The multi-agent deep reinforcement learning evaluation network is referred to as the evaluation network, and is represented by θ. The trainable parameters in the evaluation network include weight values and bias terms;

[0105] A network with the same structure as the evaluation network is built and named multi-agent deep reinforcement learning target network, referred to as target network, represented by The trainable parameters in the target network include weight values and bias terms;

[0106] The multi-agent deep reinforcement learning target network includes a space-time feature extraction layer (step 4.2.1), an individual value function fitting layer (step 4.2.2), and a global value function fitting layer (step 4.4.8);

[0107] Step 3.2, set the values of the fixed parameters required for training the evaluation network, including:

[0108] The maximum action exploration probability is ε max(Recommended value 0.95), the minimum action exploration probability is ε min (Recommended value 0.01), the number of sample items randomly extracted from the experience pool during the training process is B (recommended value 32), and the basic experience number is B * (Recommended value 400), the reward function discount factor is γ (recommended value 0.95), the experience pool size is M (recommended value 1000), the learning rate is lr (recommended value 0.001), the network update frequency is u (recommended value 10), and the maximum iteration number is J (recommended value 200).

[0109] Step 3.3, the traffic state of the intersection output by the simulation platform, the traffic state of the road network, and the reward function are taken as inputs of the multi-agent deep reinforcement learning network, and the multi-agent deep reinforcement learning network outputs individual action function values of each intersection.

[0110] The other steps and parameters are the same as one of the first to fourth embodiments.

[0111] The sixth embodiment is different from one of the first to fifth embodiments in that the step 5 repeatedly executes the step 4 to train the multi-agent deep reinforcement learning network until convergence, and a trained multi-agent deep reinforcement learning network is obtained;

[0112] The trained multi-agent deep reinforcement learning network includes:

[0113] The trained multi-agent deep reinforcement learning evaluation network and the trained multi-agent deep reinforcement learning target network;

[0114] The specific process is:

[0115] Step 4.1, setting the maximum iteration number as J;

[0116] The initial iteration number is set as 0, that is, j = 0, and θ is randomly initialized;

[0117] Step 4.2, judging whether j can be divided by u; if yes, θ is assigned to If no, the trainable parameters in the multi-agent deep reinforcement learning target network are unchanged;

[0118] The j is the iteration number, and u is the network update frequency;

[0119] The θ represents the trainable parameters in the multi-agent deep reinforcement learning evaluation network, including weight values and bias terms;

[0120] The represents the trainable parameters in the multi-agent deep reinforcement learning target network, including weight values and bias terms;

[0121] Step 4.3: Determine if the current simulation time is less than the maximum simulation time. If yes, proceed to step 4.4; otherwise, end the simulation.

[0122] Step 4.4: Obtain the trained multi-agent deep reinforcement learning network;

[0123] Step 4.4.1: Determine the traffic state of intersection i at time t. The spatiotemporal feature extraction layer (two networks; this is just to illustrate the calculation method, both networks use this method in the subsequent loss function calculation) extracts the spatiotemporal features of the intersection.

[0124] Step 4.4.2: Assign the traffic status S of the road network. t and the spatiotemporal characteristics of intersections Input the individual value function fitting layer (two networks; this is just to illustrate the calculation method, both networks use this method in the subsequent loss function calculation), and calculate the individual action function values ​​at the intersection.

[0125] Step 4.4.3: Define the set of actions for all intersections at time t. Let the joint actions of the road network at time t be denoted as .

[0126] Step 4.4.4: Set the traffic states of all intersections at time t. Defined as the joint state of the road network at time t, denoted as

[0127] Step 4.4.5, will The set of samples is stored in the experience pool as a single sample experience.

[0128] Generate a random number ε′, 0≤ε′≤1;

[0129] If ε′≤ε, the intersection i is in action A random action is selected as the set of actions for all intersections at time t. The i-th action; otherwise, choose The action corresponding to the maximum value in the value is defined as the set of actions for all intersections at time t. The i-th action; ε is the exploration probability, ε∈[ε min ,ε max ];

[0130] There are only three types of actions: increase, decrease, and remain unchanged. Indicates maintaining phase The green light display time remains unchanged; Indicates extended phase Green light display time Second; Indicates reduction of phase Green light display time Second;

[0131] Step 4.4.6: Assign t + Δt to t;

[0132] The selected action is sent to the SUMO software via the Traci traffic control interface in the SUMO software.

[0133] Step 4.4.7: Count the number of samples in the experience pool, denoted as b;

[0134] If b≤B * If the condition is not met, proceed to step 4.3; otherwise, proceed to step 4.4.8.

[0135] B * Basic experience number;

[0136] Step 4.4.8: Calculate the value function of the sample experience: The specific process is as follows:

[0137] Step 4.4.8.1: Randomly select B sample experiences from the experience pool;

[0138] Step 4.4.8.2: Based on the global value function fitting layer of the evaluation network, utilize S from sample experience. t-1 , Calculate the action function values ​​of sample experience in the evaluation network.

[0139] Step 4.4.8.3: Fit the layer based on the global value function of the target network, utilizing S from sample experience. t Calculate the action function value of the sample experience in the target network.

[0140] Step 4.4.9: Based on the action function values ​​in the target network Calculate the target label of intersection i at time t-1 in the empirical b′, denoted as b′=[1,2,...,B];

[0141]

[0142] Step 4.4.10: Calculate the loss function of the extracted B data points using formula (13);

[0143]

[0144] Step 4.4.11, update the network weight in a gradient descent manner based on the loss function, and determine whether j is less than J; if yes, execute step 4.2, and assign j+1 to j; otherwise, obtain the trained multi-agent deep reinforcement learning network.

[0145] The other steps and parameters are the same as one of the first to fifth embodiments.

[0146] The seventh embodiment is different from one of the first to sixth embodiments in that, in the step 4.4.1, the traffic state information of the intersection i at the time t is obtained by using the GAT network. The intersection spatiotemporal feature extraction layer extracts the intersection spatiotemporal feature The specific process is as follows:

[0147] Step 4.4.1.1, obtain the state information of the intersection i by using the GAT network The specific process is as follows:

[0148] Step 4.4.1.1.1, use the projection matrix W in the GAT network i t The traffic state information of the intersection i and the adjacent intersection i of the intersection i (obtained by the method described in step 2.1) is linearized (multiplication by W in formula 14), and the intersection features are sequentially spliced after linearization, and mapped to a scalar (acquired according to

[0149]

[0150] In the formula, || represents sequential splicing of features, i.e., transverse splicing of vectors;

[0151] Step 4.4.1.1.2, perform nonlinear conversion on by using the activation unit LeakyRelu(·) to obtain the attention coefficient between the target intersection at the time t and the adjacent intersection

[0152]

[0153] Step 4.4.1.1.3, normalize the attention coefficient between the intersection i and the adjacent intersection i of the intersection i by using the softmax(·) function , and obtain the state information of the intersection i in combination with the traffic state information of each adjacent intersection and the activation function σ(·)

[0154] ​​

[0155] Step 4.4.1.2, using single-layer fully connected neural network FC to convert the state information of intersection i into high-dimensional vector space to get the intermediate variable of individual value function network

[0156]

[0157] wherein, is the trainable weight parameter in the single-layer fully connected neural network;

[0158] Step 4.4.1.3, using GRU network to extract intersection spatiotemporal features The specific process is as follows:

[0159] Step 4.4.1.3.1, using intersection spatiotemporal features to represent the output result of the GRU network at time t;

[0160] Step 4.4.1.3.2, using (initial time 0) and as the input of the GRU network, and using formulas (18)-(19) to obtain the gating signals of the reset gate and the update gate

[0161]

[0162]

[0163] wherein, is the trainable weight parameter in the GRU network, is the trainable bias term in the GRU network;

[0164] Step 4.4.1.3.3, based on obtaining the intermediate variable of the GRU network by formula (20)

[0165]

[0166] wherein, is the trainable weight parameter in the GRU network, is the trainable bias term in the GRU network, and ⊙ represents pointwise multiplication between tensors;

[0167] Step 4.4.1.3.4, using and to update ​

[0168]

[0169] Step 4.4.1.4, converting the vector dimension by using the full connection layer to obtain the output result containing the intersection spatio-temporal features

[0170]

[0171] wherein, are the trainable weight parameters in the full connection layer.

[0172] The other steps and parameters are the same as one of the first to sixth embodiments.

[0173] Embodiment eight: different from one of the first to seventh embodiments, the traffic state S t and the intersection spatio-temporal features are input into the individual value function fitting layer to calculate the intersection individual action function value The specific process is as follows:

[0174] Step 4.4.2.1, the adjustment of the intersection i at the time t for the signal phase is defined as the action, denoted as

[0175] represents keeping the green light display time of the phase unchanged;

[0176] represents extending the green light display time of the phase by seconds;

[0177] represents reducing the green light display time of the phase by seconds;

[0178] Step 4.4.2.2, taking and the action at the time t-1 (when t=0, let ) as the input of the individual value function fitting layer, the individual value function fitting layer outputs the state function value V i t and the action advantage function value

[0179]

[0180]

[0181]

[0182]

[0183] where Relu(·) is an activation function; and respectively represent the output results of the intermediate network of the individual value function; is a trainable weight value in the individual value function network; is a trainable bias term in the individual value function network;

[0184] Step 4.4.2.3, calculate the individual action function value of intersection i at time t using formula (27)

[0185]

[0186] The other steps and parameters are the same as one of the first to seventh embodiments.

[0187] The ninth embodiment is different from one of the first to eighth embodiments in that the step 4.4.8, calculating the value function of the sample experience, specifically:

[0188] Step 4.4.8.1, randomly draw B sample experiences from the experience pool;

[0189] Step 4.4.8.2, calculate the action function value of the sample experience in the evaluation network based on S t-1 ,

[0190] Step 4.4.8.3, calculate the action function value of the sample experience in the target network based on S t

[0191] The step 4.4.8.2, calculating the action function value of the sample experience in the evaluation network based on S t-1 , The specific process is:

[0192] Step 4.4.8.2.1, calculate the state function value and action advantage function value V i t-1 ,

[0193] ​​​Step 4.4.8.2.2, the traffic state S of intersection i at t-1 time in the evaluation network is calculated by formula (28)-(33) in each sample experience t-1 The individual state function value of the lower intersection i and the action advantage function value

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200] In the formula, And is the calculation And The weight and bias term used in the state value function network; And is the output result of the intermediate network for calculating And The trainable weight value in the individual value function network; The trainable bias term in the individual value function network;

[0201] For the evaluation network, The action advantage function value corresponding to the actual action taken by intersection i at t-1 time in each experience Stored in the experience pool;

[0202] For the evaluation network, because there are actually selected actions in the experience pool, the value of the actually selected action is used to calculate, for example, at t-1 time, the state value function is 9, and the action advantage function values are 8, 9, and 10 (the state value function and the action advantage function value are input with the historical state in the experience, and the network structure is obtained, that is, the corresponding space-time feature extraction layer and individual value function calculation layer, t-1 time. The action actually taken at that time is recorded and calculated, the second action is actually selected in the experience pool, which is 9, Corresponding to 9, and Also calculated with 9;

[0203] Step 4.4.8.2.3, the individual state function value of intersection i at t-1 time​ the sum as a state function value in the joint state

[0204]

[0205] Step 4.4.8.2.4, calculating the action adjustment of each intersection based on S t-1 and The importance of each intersection action adjustment to the overall state change of the road network is calculated according to formulas (35)-(39)

[0206]

[0207]

[0208]

[0209]

[0210]

[0211] wherein, is the characteristic key value of intersection i; is the weight coefficient; and are intermediate variables, respectively; is the joint action of the road network at time t-1; is the trainable weight value in the individual value function network; is the trainable bias term in the individual value function network;

[0212] Step 4.4.8.2.5, calculating the action advantage function value in the joint state based on The action advantage function value in the joint state is calculated by formula (40)

[0213]

[0214] Step 4.4.8.2.6, calculating the action function value of the road network at time t-1 based on formula (41)

[0215]

[0216] The step 4.4.8.3 based on S t in the sample experience calculates the action function value of the sample experience in the target network The specific process is as follows:

[0217] Step 4.4.8.3.1: Calculate the state function value of intersection i at time t in the target network using formulas (14)-(27). Action Advantage Value With individual action function value

[0218] Step 4.4.8.3.2: Calculate the action function value of intersection i at time t in the target network using the same calculation method as in Step 4.4.8.2.

[0219] For the target network, in formula (33) yes The dominance function value corresponding to the action with the largest median value.

[0220] For example, if the state value at time t is 10, and the corresponding action advantage function values ​​are 5, 6, and 7 for three actions, then the largest action advantage function value corresponds to 7. (However, the state value and action advantage function value at time t cannot be calculated directly, so a maximization step is chosen. The state value function and the dominant action value function are calculated using the spatiotemporal feature extraction layer and the individual value function fitting layer.) For the first action, Corresponding to the second action The third action Three actions They are 8, 9, and 10 respectively, which shows the third action. The value is the largest, and the third action is the one we want to choose. The corresponding action advantage function value is 7.

[0221] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0222] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that, in step 6, the actual road network is... The traffic state at intersection i at any given time is input into a pre-trained multi-agent deep reinforcement learning evaluation network, which outputs the traffic state of intersection i at any given time. The individual action function value at time i is used to obtain the action, which is then sent to intersection i. The specific process is as follows:

[0223] Step 6.1: Use detection equipment to obtain the actual road network data. Time (first execution of command 5.1) Traffic conditions at intersection i

[0224] Step 6.2, with For the input of the trained evaluation network, output the individual action value function of intersection i at time t

[0225] Step 6.3, select the action corresponding to the maximum value in

[0226] Step 6.4, send the action to intersection i;

[0227] Step 6.5, assign the value to

[0228] Step 6.6, judge whether Δt can be divided by t; if yes, execute step 6.1; otherwise, execute step 6.5.

[0229] Other steps and parameters are the same as one of the first to ninth embodiments.

[0230] The beneficial effects of the present application are verified by the following embodiments:

[0231] Embodiment one:

[0232] Based on the real data set collected from the actual road network, the above method is compared with the traditional fixed signal timing method and six advanced deep reinforcement learning algorithms, and the experimental results are shown in Table 1. The experimental results show that the multi-agent reinforcement learning traffic signal cooperative control method considering the heterogeneity of intersections can greatly reduce the vehicle delay and parking times of the road network, and improve the vehicle travel speed and road network traffic volume.

[0233] Table 1 Comparison of performance of different deep reinforcement learning algorithms on synthetic data and real data sets

[0234]

[0235] The present application also has other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application. However, these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.​​​​​

Claims

1. A multi-agent reinforcement learning-based traffic signal cooperative control method considering the heterogeneity of intersections, characterized by: The The specific process of the method is as follows: Step 1: Data Collection; Step 2: Simulation platform setup and initialization; Step 3: Construction and initialization of multi-agent deep reinforcement learning network; Step 4: Use the traffic states of the intersections and the road network, as well as the reward function, output by the simulation platform as input to the multi-agent deep reinforcement learning network. The multi-agent deep reinforcement learning network outputs individual action function values ​​for each intersection. Based on these individual action function values, actions are determined and fed back to the simulation platform via the Traci traffic control interface in the simulation platform software. After a few seconds, output the traffic status of the intersection, the traffic status of the road network, and the reward function again; Step 5: Repeat step 4 to train the multi-agent deep reinforcement learning network until it converges, and obtain the trained multi-agent deep reinforcement learning network. A well-trained multi-agent deep reinforcement learning network includes: A trained multi-agent deep reinforcement learning evaluation network and a trained multi-agent deep reinforcement learning target network; Step 6: Place the actual road network in The traffic state at intersection i at any given time is input into a pre-trained multi-agent deep reinforcement learning evaluation network, which outputs the traffic state of intersection i at any given time. The individual action function value at time t; The optimal action is selected based on the individual action function value, and the optimal action is then sent to intersection i. The data acquisition process in step 1 is as follows: Step 1.1: Denote the intersection within the cooperative control range as i, where intersection i is... Road intersection, Let I represent the set of adjacent intersections of intersection i, and let I represent the number of intersections in the road network. Let the number of approach lanes at intersection i be denoted as . , ; Let the length of the approach lane at intersection i be denoted as . , The unit is meters. , This represents a specific entrance lane at intersection i; Let the number of exit lanes at intersection i be denoted as ; Let the length of the exit lane at intersection i be denoted as . , The unit is meters. , This represents a specific exit lane at intersection i; make This represents the number of signal phases at intersection i, and the phases are... The green light display time for each cycle is recorded as follows: , ; Step 1.2: Use variable u to represent the traffic time period. Indicates peak hours. Indicates off-peak hours; Traffic flow at intersection i during peak and off-peak hours over a consecutive week is statistically analyzed using traffic detection equipment and compiled into a real dataset. The hourly traffic volume at intersection i is denoted as... , The unit is vehicles; Step 1.3: Define the intersection phase Let F be the number of traffic flows. Calculate the intersection using the following formula. phase Key traffic flow ; (1) In the formula, , , These represent the average hourly traffic volume for traffic flows 1, 2, and F, respectively, in vehicles. This is for retrieving the maximum value. This is for the subscript operation; Will The lane being traveled is defined as ; Will Hourly traffic volume is recorded as The unit is vehicles; Intersection i and The number of lanes in the same direction of travel is denoted as ; Upstream intersection China Merge The number of lanes corresponding to the traffic flow is denoted as ; Will Downstream intersection The number of exit lanes is recorded as ; make Indicates intersection i and The distance between the upstream intersection stop lines, in meters; make Indicates an intersection and The distance between the stop lines at the downstream intersection, in meters; Step 1.4: Calculate the intersection i and i using formula (2). The heterogeneous correlation between upstream intersections is denoted as ; (2) In the formula, Indicates the scaling factor; Step 1.5: Calculate the intersection i and i using formula (3). Heterogeneous correlation between downstream intersections, denoted as ; (3) Step 2 involves the setup and initialization of the simulation platform; the specific process is as follows: The simulation platform used is SUMO software; At the start of the simulation, the environment time is set to 0, that is, set... , , This represents the maximum duration of the simulation. The data collected in Step 1 is input into the SUMO software, where it is configured as road topology files and vehicle route allocation files for different traffic periods. The SUMO software outputs the traffic status of the intersections. Traffic conditions of the road network Reward function ; The data collected in step 1 is input into the SUMO software, where it is configured as road topology files for different traffic periods. The SUMO software then outputs the traffic status of the intersection. Traffic conditions of the road network Reward function The specific process is as follows: Step 2.1: Obtain the traffic status of the intersection. The specific process is as follows: Step 2.1.1: Count the number of intersections i in the intersection. time Directional entrance lane Number of vehicles and the number of vehicles in the queue The unit is vehicles; ; Calculate the intersection i using formulas (4) and (5). time Directional entrance lane Traffic density With queue density The unit is vehicles per meter; Will The traffic density and queue density of all approach lanes at intersection i at time t constitute the traffic state of the approach lanes at intersection i at time t, denoted as . ; (4) (5) Step 2.1.2: Statistical analysis of intersection i. time Directional exit lane Number of vehicles on The unit is vehicles; The intersection i is calculated using formula (6). time Directional exit lane Remaining space index ; make The remaining space index of all exit lanes at intersection i at time t constitutes the exit lane traffic state of intersection i at time t, denoted as . ; (6) In the formula, lane The maximum number of vehicles that can be accommodated, in units of vehicles; Step 2.1.3: Set the number of the east-west straight-ahead phases to 0, and number them sequentially clockwise to count the intersections. exist The phase number of the green light is displayed at all times, denoted as... ; make The set represents the traffic state of intersection i at time t, denoted as . ; Step 2.2: Obtain the traffic status of the road network. The specific process is as follows: Step 2.2.1: Calculate the average traffic density at intersection i at time t using formula (7). The unit is vehicles per meter; The average queue density at intersection i at time t is calculated using formula (8). The unit is vehicles per meter; The combination of these two factors constitutes the average approach lane traffic state of intersection i at time t in a road network environment. ; (7) (8) Step 2.2.2: Calculate the average remaining space index of intersection i at time t using formula (9). ; At this time, the average exit lane traffic state of intersection i at time t. Equivalent to ; (9) Step 2.2.3, will The set is defined as the instantaneous traffic state of intersection i at time t. ; Step 2.2.4: Statistics begin from the initial time. Number of vehicles entering the road network during the time period Number of vehicles leaving the road network The unit is vehicles; make The set represents the traffic state of the entire road network at time t. ; Step 2.3: Obtain the reward function The specific process is as follows: Step 2.3.1: Calculate the time starting from t-1 using formula (10). Intersection during the time period Rewards for taking an action : (10) In the formula, express Starting from time t-1 Average train delay within the time period, measured in seconds (s); express Starting from time t-1 Traffic volume during a given time period, i.e., the number of vehicles that leave the stop line, expressed in vehicles; , These are the weighted values ​​corresponding to average vehicle delay and traffic volume, respectively. Negative value It is a positive value; Step 2.3.2: Calculate the global reward using formula (11). The sum of the rewards at each intersection: (11)。 2. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 1, characterized in that: Step 3 involves the construction and initialization of the multi-agent deep reinforcement learning network; the specific process is as follows: Step 3.1: Construct a multi-agent deep reinforcement learning network; The multi-agent deep reinforcement learning evaluation network includes a spatiotemporal feature extraction layer, an individual value function fitting layer, and a global value function fitting layer; Multi-agent deep reinforcement learning evaluation network, or evaluation network for short, uses... This represents the trainable parameters in a multi-agent deep reinforcement learning evaluation network, including weight values ​​and bias terms; A network with the same structure as the evaluation network is constructed and named the Multi-Agent Deep Reinforcement Learning Target Network, or simply the Target Network. This represents the trainable parameters in a multi-agent deep reinforcement learning target network, including weight values ​​and bias terms; The multi-agent deep reinforcement learning target network includes a spatiotemporal feature extraction layer, an individual value function fitting layer, and a global value function fitting layer; Step 3.2: Set the values ​​of the fixed parameters required for evaluating network training, specifically including: The maximum probability of action exploration is The probability of exploring the minimum action is The number of samples randomly drawn from the experience pool during training is B, and the base number of experiences is [number missing]. The discount factor for the reward letter is The experience pool size is M, the learning rate is lr, the network update frequency is u, and the maximum number of iterations is J; Step 3.3: Use the traffic state of the intersection, the traffic state of the road network, and the reward function output by the simulation platform as input to the multi-agent deep reinforcement learning network. The multi-agent deep reinforcement learning network outputs the individual action function values ​​of each intersection.

3. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 2, characterized in that: In step 5, step 4 is repeated to train the multi-agent deep reinforcement learning network until convergence, thus obtaining the trained multi-agent deep reinforcement learning network. A well-trained multi-agent deep reinforcement learning network includes: A trained multi-agent deep reinforcement learning evaluation network and a trained multi-agent deep reinforcement learning target network; The specific process is as follows: Step 4.1: Set the maximum number of iterations to J; The initial iteration count is set to 0, i.e., j=0, and random initialization is performed. ; Step 4.2: Determine if j divides u; if so, Assign to If not, the trainable parameters in the multi-agent deep reinforcement learning objective network. constant; j represents the iteration number, and u represents the network update frequency; The This represents the trainable parameters in a multi-agent deep reinforcement learning evaluation network, including weight values ​​and bias terms; The This represents the trainable parameters in a multi-agent deep reinforcement learning target network, including weight values ​​and bias terms; Step 4.3: Determine if the current simulation time is less than the maximum simulation time. If yes, proceed to step 4.4; otherwise, end the simulation. Step 4.4: Obtain the trained multi-agent deep reinforcement learning network; Step 4.4.1: Determine the traffic state of intersection i at time t. The spatiotemporal feature extraction layer extracts the spatiotemporal features of the intersection. ; Step 4.4.2: Assess the traffic conditions of the road network. and the spatiotemporal characteristics of intersections Input the individual value function fitting layer to calculate the individual action function values ​​at the intersection. ; Step 4.4.3: Define the set of actions for all intersections at time t. Let the joint actions of the road network at time t be denoted as . ; Step 4.4.4: Set the traffic states of all intersections at time t. Defined as the joint state of the road network at time t, denoted as ; Step 4.4.5, will The set of samples is stored in the experience pool as a single sample experience. Generate random numbers , ; if The intersection i is in action , , A random action is selected as the set of actions for all intersections at time t. The i-th action; otherwise, choose The action corresponding to the maximum value in the value is defined as the set of actions for all intersections at time t. The i-th action; It is about exploring probabilities. ; Step 4.4.6, will Assign the value to t; The selected action is sent to the SUMO software via the Traci traffic control interface in the SUMO software. Step 4.4.7: Count the number of samples in the experience pool, denoted as b; if If the condition is not met, proceed to step 4.3; otherwise, proceed to step 4.4.

8. Basic experience number; Step 4.4.8: Calculate the value function of the sample experience: The specific process is as follows: Step 4.4.8.1: Randomly select B sample experiences from the experience pool; Step 4.4.8.2: Based on the global value function fitting layer of the evaluation network, utilize the empirical data from the samples... , Calculate the action function values ​​of sample experience in the evaluation network. ; Step 4.4.8.3: Fit the target network's global value function layer, utilizing sample experience... Calculate the action function value of the sample experience in the target network. ; Step 4.4.9: Based on the action function values ​​in the target network Calculation experience Intersection i in The target label at any given time is denoted as ; ; (12) Step 4.4.10: Calculate the loss function of the extracted B data points using formula (13); (13) Step 4.4.11: Update the network weights using gradient descent based on the loss function, and determine whether j is less than J; if yes, execute step 4.2 and assign j+1 to j; otherwise, obtain the trained multi-agent deep reinforcement learning network.

4. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 3, characterized in that: In step 4.4.1, the traffic state of intersection i at time t is... The spatiotemporal feature extraction layer extracts the spatiotemporal features of the intersection. The specific process is as follows: Step 4.4.1.1: Obtain the intersection using the GAT network. Status information The specific process is as follows: Step 4.4.1.1.1: Utilize the projection matrix in the GAT network intersection and intersections Adjacent intersections Traffic status information and The features are linearized, and then concatenated sequentially to map them to a scalar. : (14) In the formula, This indicates sequential concatenation of features, i.e., horizontal concatenation of vectors; ; Step 4.4.1.1.2: Activate the unit Will Perform a nonlinear transformation to obtain the attention coefficient between the target intersection and adjacent intersections at time t. ; (15) Step 4.4.1.1.3, Utilize Function for intersection Intersection Attention coefficient between adjacent intersections Normalization is performed, and the traffic status information of each adjacent intersection is combined. With activation function Obtain the intersection Status information : (16) Step 4.4.1.2: Use a single-layer fully connected neural network to connect the intersection. Status information The intermediate variables of the individual-valued function network are obtained by uniformly transforming it to a high-dimensional vector space. ; (17) In the formula, These are the trainable weight parameters in a single-layer fully connected neural network; Step 4.4.1.3: Extract the spatiotemporal features of the intersection using a GRU network. The specific process is as follows: Step 4.4.1.3.1: Use the spatiotemporal characteristics of the intersection This represents the output of the GRU network at time t; Step 4.4.1.3.2, using time t-1 as an example. as well as For the input of the GRU network, use formulas (18)-(19) to obtain the gate control signals of the reset gate and update gate. , ; (18) (19) In the formula, , , , These are the trainable weight parameters in a GRU network. , It is a trainable bias term in the GRU network; Step 4.4.1.3.3, based on Obtain the intermediate variables of the GRU network from formula (20) : (20) In the formula, , These are the trainable weight parameters in a GRU network. It is a trainable bias term in the GRU network. This represents pointwise multiplication between tensors; Step 4.4.1.3.4, utilizing and accomplish Update: (21) Step 4.4.1.4: Use the fully connected layer to transform the vector dimension to obtain the output result containing the spatiotemporal features of the intersection. : (22) In the formula, These are the trainable weight parameters in a fully connected layer.

5. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 4, characterized in that: In step 4.4.2, the traffic status of the road network will be... and the spatiotemporal characteristics of intersections Input the individual value function fitting layer to calculate the individual action function values ​​at the intersection. The specific process is as follows: Step 4.4.2.1: Assume the signal phase of intersection i at time t. The adjustment is defined as an action, denoted as ; Indicates maintaining phase The green light display time remains unchanged; Indicates extended phase Green light display time Second; Indicates reduction of phase Green light display time Second; Step 4.4.2.2, with Actions at time t-1 ( season The input to the individual value function fitting layer is given by formulas (23)-(26), and the output of the individual value function fitting layer is given by formulas (23)-(26). The state function value of the intersection at any time With action advantage function value : (23) (24) (25) (26) In the formula, It is an activation function; and These represent the output results of the intermediate network of the individual value function; , , , These are the trainable weights in an individual value function network; , , , It is a trainable bias term in an individual value function network; Step 4.4.2.3: Calculate the individual action function value of intersection i at time t using formula (27). ; (27)。 6. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 5, characterized in that: The specific process for calculating the value function of the sample experience in step 4.4.8 is as follows: Step 4.4.8.1: Randomly select B sample experiences from the experience pool; Step 4.4.8.2, based on sample experience , Calculate the action function values ​​of sample experience in the evaluation network. ; Step 4.4.8.3, based on sample experience Calculate the action function value of the sample experience in the target network. ; In step 4.4.8.2, based on sample experience... , Calculate the action function values ​​of sample experience in the evaluation network. The specific process is as follows: Step 4.4.8.2.1: In the evaluation network, calculate the intersection i in each sample experience using formulas (14)-(26). State function value and action advantage function value at time -1 , ; Step 4.4.8.2.2: In the evaluation network, calculate the intersection i in each sample experience using formulas (28)-(33). Traffic conditions of the road network at time -1 Individual state function value and action advantage function value at intersection i , : (28) (29) (30) (31) (32) (33) In the formula, and It is a calculation and The weights and biases used; and It is a calculation and The output of the intermediate network; , , , These are the trainable weights in an individual value function network; , , , It is a trainable bias term in an individual value function network; For evaluating networks, It represents the actual actions taken by intersection i at time t-1, stored in each set of experiences. The corresponding dominant function value; Step 4.4.8.2.3, with The individual state function value of intersection i at time -1 The sum of these values ​​serves as the state function value under the joint state. : (34) Step 4.4.8.2.4, based on as well as Calculate the importance of the operation adjustment of each intersection to the overall state of the road network according to formulas (35)-(39). : (35) (36) (37) (38) (39) In the formula, It is the feature key value of intersection i; These are weighting coefficients; and These are intermediate variables; For the road network in Joint actions at time -1; , , , These are the trainable weights in an individual value function network; , , , It is a trainable bias term in an individual value function network; Step 4.4.8.2.5, based on The action advantage function value under the joint state is calculated using formula (40). ; (40) Step 4.4.8.2.6: Calculate the action function value of the road network at time t-1 based on formula (41). ; (41) In step 4.4.8.3, based on sample experience... Calculate the action function value of the sample experience in the target network. The specific process is as follows: Step 4.4.8.3.1: Calculate the intersection i in the empirical data of each sample in the target network using formulas (14)-(27). State function value at time 1 Action advantage value With individual action function value ; Step 4.4.8.3.2: In the target network, calculate the intersection i in each sample's experience using the same calculation method as in Step 4.4.8.

2. Action function value at time 1 ; For the target network, in formula (33) yes The dominance function value corresponding to the action with the largest median value.

7. The multi-agent reinforcement learning traffic signal cooperative control method considering intersection heterogeneity according to claim 6, characterized in that: In step 6, the actual road network is... The traffic state at intersection i at any given time is input into a pre-trained multi-agent deep reinforcement learning evaluation network, which outputs the traffic state of intersection i at any given time. The individual action function value at time i is used to obtain the action, which is then sent to intersection i. The specific process is as follows: Step 6.1: Use detection equipment to obtain the actual road network data. Traffic conditions at intersection i at any time ; Step 6.2, with As input to the trained evaluation network, the output intersection i is... Individual action value function at time 1 ; Step 6.3, Select Action corresponding to the maximum value ; Step 6.4, Issued to intersection i; Step 6.5, +1 is assigned to ; Step 6.6, Judgment Is it divisible? If yes, proceed to step 6.1; otherwise, proceed to step 6.5.

Images