Traffic signal control method and device
Through the multi-agent deep reinforcement learning model based on attention mechanism and combined with SAC algorithm, the problem of information extraction and spatial search in trunk traffic signal control is solved, the traffic signal control efficiency is improved, and the traffic network traffic management is achieved is achieved.
Patent Information
- Application Number
- CN202210880024.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-07-25
AI Technical Summary
The existing multi-agent deep reinforcement learning model is difficult to effectively extract information and search for optimal solution space in trunk traffic signal control, resulting in low traffic signal control efficiency.
Using a multi-agent deep reinforcement learning model based on attention mechanism, the pre-trained ALight network and ACritic neural network are combined with SAC deep reinforcement learning algorithm to obtain the road network environment information and traffic flow information of the signal light, determine the phase of the signal light and perform the Control and improve the traffic efficiency of trunk line network.
It significantly improves the traffic efficiency of the trunk line network, enhances the search ability of the MARL model for solution space, and extracts effective information through attention mechanism to provide better traffic signal control.
Smart Images

Figure CN115359653B_ABST
Abstract
Description
Technical Field
[0001] This article relates to the field of deep learning technology, and more particularly to a traffic signal control method and device. Background Art
[0002] Traffic signal control is crucial to building smart cities. With the development of deep reinforcement learning (DRL) technology, a large number of studies have applied the DRL method to single-intersection traffic signal control. In recent years, researchers have begun to apply the DRL method to trunk traffic signal control and multi-intersection traffic signal control. For trunk traffic signal control, since the state, i.e., action space, of the trunk traffic signal control problem is huge, it is necessary to design a multi-agent deep reinforcement learning (MARL) model to extract useful information and search for the optimal solution space. The design of the MARL model is mainly affected by three factors: the selection of reinforcement learning model elements, the design of the multi-agent reinforcement learning model structure, and the multi-agent model training algorithm, so there are certain difficulties in the design of this model.
[0003] Therefore, there is an urgent need to implement a multi-agent deep reinforcement learning trunk traffic signal control method based on the attention mechanism to achieve effective control of trunk traffic signals. Summary of the invention
[0004] The present application provides a traffic signal control method and device, which determines traffic signals based on a multi-agent deep reinforcement learning traffic signal control model with an attention mechanism, and can improve the traffic efficiency of trunk road networks.
[0005] The present application provides a traffic signal control method, the method comprising:
[0006] Obtain the road network environment information of the traffic lights and the current traffic flow information;
[0007] Determining intersection observation information corresponding to the signal light according to the road network environment information and the current traffic flow information;
[0008] Determine the phase of the traffic light at the current moment by using the pre-trained ALight network of the multi-agent deep reinforcement learning traffic signal control model based on the attention mechanism, and control the traffic light according to the determined phase;
[0009] The input parameters of the traffic signal control model include: intersection observation information corresponding to the traffic light, and the output parameter of the traffic signal control model is the phase of the traffic light at the current moment.
[0010] In an exemplary embodiment, the pre-trained traffic signal control model based on the attention mechanism is trained by the following steps:
[0011] Establish an experience replay pool, which stores multiple sets of data, each set of data includes: current state information, current signal light phase selection, reward, current state information; wherein the state information includes: intersection observation information of signal lights, road network global observation information;
[0012] A batch of data is randomly selected from the experience replay pool for this round of traffic signal control model training;
[0013] The model obtained from this round of training is stored;
[0014] After a predetermined number of rounds of training, the model with the largest reward in the rounds of training is selected as the trained traffic signal control model.
[0015] In an exemplary embodiment, establishing an experience replay pool includes:
[0016] Generate road network files through preset simulation software and road network environment information;
[0017] Generate a road network traffic flow file based on the driving path of each vehicle in the road network;
[0018] Generate a simulation configuration file according to the road network file and the road network traffic flow file, and perform traffic signal control simulation according to the simulation configuration file;
[0019] The sample data of each round of simulation is stored in the experience replay pool.
[0020] In an exemplary embodiment, randomly selecting a batch of data from the experience playback pool for current round of traffic signal control model training includes:
[0021] A batch of data is randomly selected from the experience replay pool, and the pre-established attention-based ACritic neural network model is trained and updated;
[0022] The reward Q value is calculated based on the selected batch of data and the ACritic neural network model, and the pre-established ALight neural network model based on the attention mechanism is updated;
[0023] Through the updated ALight neural network model, the preset deep reinforcement learning algorithm is used for calculation to obtain the current phase of the traffic light, completing the training of the traffic signal control model.
[0024] In an exemplary embodiment, the ACritic neural network model is:
[0025]
[0026] In the above model, wi represents the parameters of the Acritic model, i represents the current agent, and i is a positive integer greater than or equal to 1; the input information of the Acritic neural network model is the global observation information of the road network S t , t represents the current moment, and the output information of the Acritic neural network model is the cumulative reward of each phase selected by the agent.
[0027] In an exemplary embodiment, the ALight neural network model is:
[0028]
[0029] In the above model, θ i represents the parameters of the ALight neural network model, softmax(·) is the activation function, ||L i || and ||p i || respectively represent the number of lanes and the number of intersection phases at the intersection where the current agent i is located; the input information of the ALight neural network model is the intersection observation information o i,t ; The output information of the ALight neural network model is the probability of the agent selecting each phase.
[0030] In an exemplary embodiment, the preset deep reinforcement learning algorithm includes:
[0031]
[0032] The goal of this algorithm is is to maximize the cumulative reward and maximize the entropy, T represents the time of one round of training model, α represents the temperature parameter, r i,t Represents the reward obtained by the signal light at time t.
[0033] In an exemplary embodiment, the reward Q value is calculated based on the global observation information of the road network in the data and the ACritic neural network model, and the reward Q value is used to update the pre-established ALight neural network model based on the attention mechanism, including:
[0034] Use the update formula of the ACritic model to calculate the reward Q value;
[0035] Calculate the ALight neural network model according to the reward Q value calculated by the ACritic model;
[0036] Among them, the update formula of the ACritic model is:
[0037]
[0038] Where D represents the experience replay pool, represents the reward Q value calculated by the ACritic network, y(r i,t ,S t+1 ) represents the target reward Q value, a t represents the temperature parameter, (S t , a t )~D means randomly selecting data from the experience pool, and E(.) means averaging the data.
[0039] In an exemplary embodiment, the update formula of the ALight model is:
[0040]
[0041] in, Represents the reward Q value calculated using the ACritic model.
[0042] The present application also provides a traffic signal control device, which includes: a memory and a processor; the memory is used to store a program for performing traffic signal control, and the processor is used to read and execute the program for performing traffic signal control, and execute any one of the methods described in the above embodiments.
[0043] Compared with the related art, the present application provides a traffic signal control method and device, the method comprising: obtaining the road network environment information of the traffic light and the traffic flow information at the current moment; determining the intersection observation information corresponding to the traffic light according to the road network environment information and the traffic flow information at the current moment; using a pre-trained traffic signal control model based on the attention mechanism to determine the phase of the traffic light at the current moment, and controlling the traffic light according to the determined phase; wherein the input parameters of the traffic signal control model include: the intersection observation information corresponding to the traffic light, and the output parameters of the traffic signal control model are the phase of the traffic light at the current moment. Through the technical solution of the present invention, the method determines the traffic signal based on the traffic signal control model of the attention mechanism, which can improve the traffic efficiency of the trunk road network.
[0044] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.
[0046] Figure 1 A flow chart of a traffic signal control method according to an embodiment of the present application;
[0047] Figure 2 A schematic diagram of a traffic signal control device according to an embodiment of the present application;
[0048] Figure 3 A schematic diagram of establishing and training a traffic signal control model in some exemplary embodiments;
[0049] Figure 4 A schematic diagram of a 1×6 trunk line network tested in some exemplary embodiments;
[0050] Figure 5 A schematic diagram of the network structure of a trunk traffic signal control method of a multi-agent deep reinforcement learning traffic signal control model based on an attention mechanism in some exemplary embodiments;
[0051] Figure 6 A schematic diagram of the network structure of the ALight model in some exemplary embodiments;
[0052] Figure 7 Schematic diagram of the ACritic model network structure in some exemplary embodiments. DETAILED DESCRIPTION
[0053] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.
[0054] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.
[0055] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.
[0056] The present disclosure provides a traffic signal control method. Figure 1 As shown, the method includes steps S100-S120, which are specifically as follows:
[0057] S100. Obtaining signal light network environment information and current traffic flow information;
[0058] S110. Determine the intersection observation information and the road network global observation information corresponding to the signal light according to the road network environment information and the current traffic flow information;
[0059] S120. Determine the phase of the traffic light at the current moment using a pre-trained traffic signal control model based on the attention mechanism, and control the traffic light according to the determined phase.
[0060] In this embodiment, the input parameters of the traffic signal control model include: intersection observation information corresponding to the traffic light, and the output parameter of the traffic signal control model is the phase of the traffic light at the current moment.
[0061] In step S100, the road network environment information includes but is not limited to: the number of lanes in the road section and the phase setting of the intersection. Figure 4 As shown: the number of lanes on the road section is 3; the intersection phase settings include: phase 1, phase 2, phase 3 and phase 4. The traffic flow information includes the number of vehicles at each intersection under the current traffic light and the driving path of each vehicle.
[0062] In step S110, each traffic light can only obtain its own intersection observation information. i,t , the information includes: the number of vehicles in each lane n i,t , the average speed of vehicles in each lane v i,t and the current phase p of the intersection i,t The intersection observation information can be expressed as (||L i ||,2+||pi||), where ||Li || and ||p i || respectively represent the number of lanes and the number of intersection phases. The global observation information of the road network is represented by S t , which can be the global observation information of the trunk network or the global observation information of the non-trunk network, using (N,||L i ||,2+||p i ||) represents the observation information and of all intersections in the road network.
[0063] In an exemplary embodiment, the pre-trained traffic signal control model based on the attention mechanism is trained by the following steps:
[0064] Step 1. Establish an experience replay pool, which stores multiple sets of data, each set of data includes: current state information, current signal light phase selection, reward, current state information; wherein the state information includes: intersection observation information of signal lights, road network global observation information;
[0065] Step 2. Randomly select a batch of data from the experience playback pool for this round of traffic signal control model training;
[0066] Step 3. The result model obtained from this round of training is stored;
[0067] Step 4. After a predetermined number of rounds of training, the model with the largest reward in the multiple rounds of training is selected as the trained traffic signal control model.
[0068] In an exemplary embodiment, establishing an experience replay pool includes:
[0069] Generate road network files through preset simulation software and road network environment information;
[0070] Generate a road network traffic flow file based on the driving path of each vehicle in the road network;
[0071] Generate a simulation configuration file according to the road network file and the road network traffic flow file, and perform traffic signal control model training according to the simulation configuration file;
[0072] The results of each round of model training are stored in the experience replay pool.
[0073] In an exemplary embodiment, randomly selecting a batch of data from the experience playback pool for current round of traffic signal control model training includes:
[0074] A batch of data is randomly selected from the experience replay pool, and the pre-established attention-based ACritic neural network model is trained and updated;
[0075] The reward Q value is calculated based on the selected batch of data and the ACritic neural network model, and the pre-established ALight neural network model of multi-agent deep reinforcement learning based on the attention mechanism is updated;
[0076] Through the updated ALight neural network model, the preset deep reinforcement learning algorithm is used for calculation to obtain the current phase of the traffic light, completing the training of the traffic signal control model.
[0077] In an exemplary embodiment, the ACritic neural network model is:
[0078]
[0079] In the above model, w i represents the parameters of the Acritic model, i represents the current agent, and i is a positive integer greater than or equal to 1; the input information of the Acritic neural network model is the global observation information of the road network S t , t represents the current moment, and the output information of the Acritic neural network model is the cumulative reward of each phase selected by the agent.
[0080] In an exemplary embodiment, the ALight neural network model is:
[0081]
[0082] In the above model, θ i represents the parameters of the ALight neural network model, softmax(·) is the activation function, ||L i || and ||p i || respectively represent the number of lanes and the number of intersection phases at the intersection where the current agent i is located; the input information of the ALight neural network model is the intersection observation information o i,t ; The output information of the ALight neural network model is the probability of the agent selecting each phase.
[0083] In an exemplary embodiment, the preset deep reinforcement learning algorithm includes:
[0084]
[0085] The goal of this algorithm is is to maximize the cumulative reward and maximize the entropy, T represents the time of one round of training model, α represents the temperature parameter, r i,t Represents the reward obtained by the signal light at time t.
[0086] In an exemplary embodiment, the reward Q value is calculated based on the global observation information of the road network in the data and the ACritic neural network model, and the reward Q value is used to update the pre-established ALight neural network model based on the attention mechanism, including:
[0087] Use the update formula of the ACritic model to calculate the reward Q value;
[0088] Calculate the ALight neural network model according to the reward Q value calculated by the ACritic model;
[0089] Among them, the update formula of the ACritic model is:
[0090]
[0091] Where D represents the experience replay pool, represents the reward Q value calculated by the ACritic network, y(r i,t ,S t+1 ) represents the target reward Q value, a t represents the temperature parameter, (S t , a t )~D means randomly selecting data from the experience pool, and E(.) means averaging the data.
[0092] In an exemplary embodiment, the update formula of the ALight model is:
[0093]
[0094] in, Represents the reward Q value calculated using the ACritic model.
[0095] The present disclosure also provides a traffic signal control device, such as Figure 2 As shown, the device includes: a memory 210 and a processor 220; the memory 210 is used to store the program for performing traffic signal control, and the processor 220 is used to read and execute the program for performing traffic signal control, and execute any one of the methods in the above embodiments.
[0096] The following example illustrates the establishment and training process of the traffic signal control model. Figure 3 As shown:
[0097] Step 301: Building a road network environment;
[0098] Generate traffic flow files and simulation configuration information files according to the constructed road network environment;
[0099] Step 302: Setting Markov decision process elements;
[0100] The arterial traffic signal control is modeled as a partially observable Markov decision process, and the elements of the arterial traffic signal control Markov decision process are set;
[0101] Step 303: Building a multi-agent deep reinforcement learning traffic signal control model based on the attention mechanism;
[0102] Step 304: Using SAC (Soft actor critic) deep reinforcement learning algorithm to train the traffic signal control model;
[0103] Step 305: Define traffic signal control evaluation indicators and test the control effect of trunk traffic signals.
[0104] In step 301, the process of generating a simulation configuration file is as follows:
[0105] Step 3011, using the SUMO microscopic traffic simulation software, modeling the intersection channelization according to the topology of the trunk road network, including: node generation, road section lane number definition, edge generation, road section connectivity setting, intersection phase setting, detector generation. Finally, a .net.xml file is generated;
[0106] Step 3012: According to each vehicle veh i At time t i From the road arc j Enter, on the main road network driving path arc i,j →arc i,j+1 →…, then leave the road network information and generate the trunk road network traffic flow .rou.xml file;
[0107] Step 3013, generate a simulation configuration .sumocfg file, the configuration information includes: simulation road network .net.xml file, simulation traffic .rou.xml file, road network detector .add.xml file, simulation start and end time, and whether to visualize the simulation.
[0108] In step 302, the process of setting the elements of the Markov decision process for arterial traffic signal control is as follows:
[0109] Step 3021: Each signal light obtains observation information of its own intersection. i,t , including: the number of vehicles in each lane n i,t , the average speed of vehicles in each lane v i,t and the current phase p of the intersection i,t The intersection observation information is represented as (||L i||,2+||p i ||) vector, where ||L i || and ||p i || respectively represent the number of lanes and the number of intersection phases;
[0110] Step 3022, state S t Represented as the global observation information of the trunk line network, using (N,||L i ||,2+||p i ||) tensor represents the observation information of all intersections in the road network;
[0111] Step 3023: The action of each signal light is set to phase selection type, which is represented as in, Indicates the pre-set phase scheme of the intersection. The current phase selection action setting indicates the current intersection release phase a i,t Vehicles in the lane;
[0112] Step 3024: Reward function r of each traffic light i,t Set to the average number of vehicles queued in the lanes of each intersection of the signal lights.
[0113] In step 303, a multi-intelligence deep reinforcement learning model based on the attention mechanism is built. The specific method is:
[0114] Step 3031: Establish an actor network model based on the attention mechanism, namely the ALight neural network model. The ALight neural network model can more effectively extract effective information from intersections. In order to more effectively extract features, the ALight neural network model uses an embed layer with an output of 128 dimensions. In order to capture the importance of each embed phase feature, the attention mechanism is used. The input of the ALight neural network model is the intersection observation information. i,t , the dimension is (||L i ||,2+||pi||); the output information of the ALight neural network model is the probability of the agent choosing each phase, which is (1,||p i ||) vector;
[0115] like Figure 6 As shown, the ALight neural network model is:
[0116]
[0117] In the above formula, θ i represents the network parameters of the ALight model, and softmax(·) is the activation function.
[0118] Step 3032: Establish a critic network model based on the attention mechanism, namely, an Acritic neural network model. The Acritic neural network model estimates the Q function more accurately and can reduce the Actor network gradient estimation error. In order to more effectively extract intersection features, the Acritic neural network model uses an embed layer with an output of 128 dimensions. In order to capture the importance of each intersection embed feature, an attention mechanism is used. The input of the Acritic model is the global observation information S of the road network. t , the dimension is (N,||L i ||,2+||p i ||), the output information of the Acritic neural network model is the cumulative reward of the agent for each phase, which is (1, ||p i ||) vector;
[0119] Among them, Figure 7 As shown, the ACritic neural network model is:
[0120]
[0121] In the above formula, w i Represents the network parameters of the ACritic model.
[0122] In step 304, the SAC deep reinforcement learning algorithm is used as the training algorithm of the multi-agent model. The SAC deep reinforcement learning algorithm is:
[0123]
[0124] Where T represents the simulation time range, and α is the temperature parameter that controls the importance of the entropy term to the reward. SAC is used as the training algorithm for the multi-agent reinforcement learning model, with the goal of maximizing the cumulative reward while also maximizing entropy.
[0125] In step 304, the model training process includes updating the ACritic neural network model and the ALight neural network model;
[0126] Among them, the ACritic neural network model update formula is expressed as:
[0127]
[0128] Where D represents the experience replay pool, represents the reward Q value calculated by the ACritic network, y(r i,t ,S t+1 ) represents the target reward Q value, a t represents the temperature parameter, (St , a t )~D represent randomly selecting data from the experience pool, and E(.) represents averaging the data;
[0129] The ALight model neural network update formula is expressed as:
[0130]
[0131] In the above update formula, represents the reward Q value calculated using the ACritic model;
[0132] The temperature parameter gradient update formula is expressed as:
[0133]
[0134] In step 305, the effect evaluation process of the trained traffic signal control model is as follows:
[0135] (1) In order to evaluate the training performance of various algorithms, the best convergence rounds (BCE) and best convergence time (BCT) required for the training algorithm are used. The BCE indicator can reflect the sample efficiency of the algorithm, and the BCT indicator reflects the time complexity of the algorithm. Both evaluation indicators are obtained through the best training model in all training experiments. In addition, the best convergence point (BCP) of the average round reward curve is also used to evaluate the training effect of the reinforcement learning algorithm.
[0136] (2) In order to evaluate the execution performance of various reinforcement learning algorithms, the average round cumulative reward (ACR), average queue length (AQL), average waiting time (AWT), average travel time (ATT), and average number of stops (AS) are used as evaluation indicators in the execution stage.
[0137] (3) The comparison algorithm uses the max band method and several multi-agent deep reinforcement learning algorithms, CommNet and NeurComm models.
[0138] Based on the above comparative analysis, it is found that the traffic signal control model based on the attention mechanism in this embodiment has the following significant advantages compared with the MARL model:
[0139] 1. Using SAC as the training algorithm of the MARL model greatly enhances the MARL model's ability to search the solution space;
[0140] 2. The attention mechanism is used in the actor and critic neural network models, which enhances the ability of the MARL model to extract effective information;
[0141] 3. The established MARL model based on the attention mechanism provides important reference value for the model design of other MARL problems.
[0142] The following is an example to illustrate the process of the traffic signal control method. Figure 5 As shown:
[0143] Step 501. Generate a road network file through preset simulation software and road network environment information;
[0144] In this step, you can use Sumo simulation software to generate Figure 4 The road network file 1×6_grid.net.xml shown in Figure 4 The road network is a 1×6 trunk road network with three lanes in both directions and a section length of 300m. The traffic lights at each intersection are set with four phases, including north-south straight, north-south left turn, east-west straight and east-west left turn.
[0145] Step 502: Generate a road network traffic flow file according to the driving path of each vehicle in the road network;
[0146] In this step, Sumo simulation software can be used to generate trunk road network traffic flow based on vehicle trajectory information. For example, the trajectory information of each vehicle on the trunk road network is: veh i At time t i From the road arc j Enter, the driving path on the trunk road network is arc i,j →arc i,j+1 →…, and finally leave the road network; generate the trunk road network traffic flow .rou.xml file based on the trajectory information.
[0147] Step 503: Generate a simulation configuration file according to the road network file and the road network traffic flow file;
[0148] In this step, you can use Sumo simulation software to generate a simulation configuration .sumocfg file. The configuration information includes: simulation road network 1×6_grid.net.xml, simulation traffic 1×6_grid.rou.xml, road network detector 1×6_grid.add.xml, simulation start time = 0, end time = 3600s, and whether to visualize the simulation = False.
[0149] Step 504: Determine the intersection observation information and the road network global observation information corresponding to the signal light according to the road network environment information and the traffic flow information at the current moment;
[0150] In this step, the partially observable Markov decision process elements of the trunk traffic signal control model include: intersection observation information of the signal light and global observation information of the road network. Among them, each signal light obtains the observation information of its own intersection. i,t , including: the number of vehicles in each lane n i,t , the average speed of vehicles in each lane v i,t and the current phase p of the intersection i,t The intersection observation information is represented as (||L i ||,2+||p i ||) vector, where ||L i || and ||p i || respectively represent the number of lanes and the number of intersection phases. State S t Represented as the global observation information of the trunk line network, using (N,||L i ||,2+||p i ||) represents the observation information and of all intersections in the road network.
[0151] The elements of the partially observable Markov decision process modeling of trunk traffic signal control also include: the action of each signal light is set to phase selection type, expressed as in Indicates the pre-set phase scheme of the intersection: north-south straight, north-south left turn, east-west straight, east-west left turn. Phase selection action setting indicates the current intersection release phase a i,t Vehicles in the lane;
[0152] The elements of the partially observable Markov decision process modeling of arterial traffic signal control also include: the reward value of each signal light is set to the average number of queued vehicles in the lanes of the respective intersections.
[0153] Step 505. Obtain a pre-trained traffic signal control model based on the attention mechanism, and determine the phase of the traffic light at the current moment.
[0154] In this step, a multi-intelligence deep reinforcement learning model based on the attention mechanism is built, including:
[0155] Step 5051. Establish an experience replay pool, which stores multiple groups of data, each group of data includes: current state information, current signal light phase selection, rewards, current state information; wherein, the state information includes: intersection observation information of signal lights, global observation information of the road network.
[0156] In this step, an experience replay pool is established, including:
[0157] 1. Generate road network files through preset simulation software and road network environment information;
[0158] 2. Generate a road network traffic flow file based on the driving path of each vehicle in the road network;
[0159] 3. Generate a simulation configuration file according to the road network file and the road network traffic flow file, and perform traffic signal control model training according to the simulation configuration file;
[0160] 4. Store the results of each round of model training in the experience replay pool.
[0161] Step 5052. Randomly select a set of data from the experience replay pool and obtain a pre-established attention mechanism-based ACritic neural network model; Figure 7 As shown, the ACritic neural network model is:
[0162]
[0163] In the above model, w i represents the parameters of the Acritic model, i represents the current agent, and i is a positive integer greater than or equal to 1; the input information of the Acritic neural network model is the global observation information of the road network S t , t represents the current moment, and the output information of the Acritic neural network model is the cumulative reward of each phase selected by the agent.
[0164] Step 5053. Calculate a reward Q value based on the global observation information of the road network in the data and the ACritic neural network model, and use the reward Q value to update the pre-established ALight neural network model based on the attention mechanism;
[0165] The implementation process in this step is as follows:
[0166] Step 01. Pre-establish an actor network based on the attention mechanism to more effectively extract effective information from intersections. This actor network is called ALight. Figure 6 As shown, the ALight neural network model is:
[0167]
[0168] In the above model, θ i represents the parameters of the ALight neural network model, softmax(·) is the activation function, ||L i || and ||p i|| respectively represent the number of lanes and the number of intersection phases at the intersection where the current agent i is located; the input information of the ALight neural network model is the intersection observation information o i,t ; The output information of the ALight neural network model is the probability of the agent selecting each phase.
[0169] Step 02. Use the SAC deep reinforcement learning algorithm as the training algorithm for the traffic signal control model. The specific method is as follows:
[0170] The preset deep reinforcement learning algorithm includes:
[0171]
[0172] The goal of this algorithm is is to maximize the cumulative reward and maximize the entropy, T represents the time of one round of training model, α represents the temperature parameter, r i,t Represents the reward obtained by the signal light at time t.
[0173] Step 03: Calculate the reward Q value using the update formula of the ACritic model; the update formula of the ACritic model is:
[0174]
[0175] Where D represents the experience replay pool, represents the reward Q value calculated by the Acritic network, y(r i,t ,S t+1 ) represents the target reward Q value, a t represents the temperature parameter, (S t , a t )~D means randomly selecting data from the experience pool, and E(.) means averaging the data.
[0176] Step 04: After calculating the reward Q value according to the update formula of the ACritic model, update and calculate the ALight neural network model;
[0177] The update formula of the ALight model is:
[0178]
[0179] in, Represents the reward Q value calculated using the ACritic model.
[0180] Step 05: Calculate the updated ALight model using the updated temperature parameters;
[0181] Among them, the temperature parameter gradient update formula in the algorithm is expressed as:
[0182]
[0183] Step 506: Evaluate the effectiveness of the traffic signal control method.
[0184] In this step, 1×6_grid.rou.xml traffic is used for simulation to test the training effect of the traffic signal control model based on the attention mechanism, including CommNet and NeurComm models.
[0185] In order to evaluate the training performance of various algorithms, the best number of convergence rounds (BCE) and best convergence time (BCT) required for the training algorithm were used;
[0186] Use 1×6_grid.rou.xml traffic to test the algorithm model trained in step 4 and compare it with the maxband method and CommNet and NeurComm models;
[0187] In order to evaluate the execution performance of various reinforcement learning algorithms, the average cumulative reward (ACR), average queue length (AQL), average waiting time (AWT), average travel time (ATT) and average number of stops (AS) are used as evaluation indicators in the execution stage.
[0188] In this embodiment, through the traffic signal control model based on the attention mechanism, useful information can be extracted and the optimal solution space can be searched. The model uses SAC as the training algorithm of the MARL model, which greatly enhances the MARL model's search ability for the solution space; the model uses the attention mechanism in the actor and critic neural network models, which enhances the MARL model's ability to extract effective information, and can also provide better control signals for trunk traffic to improve the traffic efficiency of the trunk road network.
[0189] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
Claims
1. A traffic signal control method, characterized in that: The method comprises: Obtain the road network environment information of the traffic lights and the current traffic flow information; Determining intersection observation information corresponding to the signal light according to the road network environment information and the current traffic flow information; Determine the phase of the traffic light at the current moment by using a pre-trained multi-agent deep reinforcement learning traffic signal control model based on an attention mechanism, and control the traffic light according to the determined phase; The input parameters of the traffic signal control model include: intersection observation information corresponding to the signal light, and the output parameter of the traffic signal control model is the phase of the signal light at the current moment; The pre-trained multi-agent deep reinforcement learning traffic signal control model based on the attention mechanism is trained by the following steps: Establish an experience replay pool, which stores multiple sets of data, each set of data includes: current state information, current signal light phase selection, reward, current state information; wherein the state information includes: intersection observation information of signal lights, road network global observation information; A batch of data is randomly selected from the experience replay pool for this round of traffic signal control model training, including: randomly selecting a batch of data from the experience replay pool, and training and updating a pre-established AClitic neural network model based on an attention mechanism; calculating a reward Q value based on the selected batch of data and the AClitic neural network model, and updating a pre-established ALight neural network model based on an attention mechanism; using the updated ALight neural network model, using a preset deep reinforcement learning algorithm to calculate, obtain the current phase of the traffic light, and complete one training of the traffic signal control model; The model obtained from this round of training is stored; After a predetermined number of rounds of training, the model with the largest reward in the rounds of training is selected as the trained traffic signal control model.
2. The traffic signal control method according to claim 1, characterized in that: The establishing of the experience replay pool includes: Generate road network files through preset simulation software and road network environment information; Generate a road network traffic flow file based on the driving path of each vehicle in the road network; Generate a simulation configuration file according to the road network file and the road network traffic flow file, and perform traffic signal control simulation according to the simulation configuration file; The sample data of each round of simulation is stored in the experience replay pool.
3. The traffic signal control method according to claim 1, characterized in that: The ACritic neural network model is: In the above model, w i represents the parameters of the Acritic model, i represents the current agent, and i is a positive integer greater than or equal to 1; the input information of the Acritic neural network model is the global observation information of the road network S t , t represents the current moment, and the output information of the Acritic neural network model is the cumulative reward of the agent for each phase.
4. The traffic signal control method according to claim 3, characterized in that: The ALight neural network model is: In the above model, θ i represents the parameters of the ALight neural network model, softmax(·) is the activation function, L i and p i They represent the number of lanes and the number of intersection phases at the intersection where the current agent i is located; the input information of the ALight neural network model is the intersection observation information o i,t ;The output information of the ALight neural network model is the probability of the agent selecting each phase.
5. The traffic signal control method according to claim 4, characterized in that: The preset deep reinforcement learning algorithm includes: The goal of this algorithm is is to maximize the cumulative reward and maximize the entropy, T represents the time of one round of training model, α represents the temperature parameter, r i,t Represents the reward obtained by the signal light at time t.
6. The traffic signal control method according to claim 4, characterized in that: The reward Q value is calculated based on the global observation information of the road network in the data and the ACritic neural network model, and the reward Q value is used to update the pre-established ALight neural network model based on the attention mechanism, including: Use the update formula of the ACritic model to calculate the reward Q value; Calculate the ALight neural network model according to the reward Q value calculated by the ACritic model; Among them, the update formula of the ACritic model is: Where D represents the experience replay pool, represents the reward Q value calculated by the ACritic network, y(r i,t ,S t+1 ) represents the target reward Q value, at represents the temperature parameter, (St, at)~D represents randomly selecting data from the experience pool, and E(.) represents averaging the data.
7. The traffic signal control method according to claim 6, characterized in that: The update formula of the ALight model is: in, Represents the reward Q value calculated using the ACritic model.
8. A traffic signal control device, characterized in that: The device comprises: a memory and a processor; characterized in that the memory is used to store a program for performing traffic signal control, and the processor is used to read and execute the program for performing traffic signal control, and execute the method described in any one of claims 1-7.
