High-adaptability road network level traffic signal control method capable of complementing missing observation in real time

Abstract

The invention discloses a high-adaptability road network level traffic signal control method capable of complementing missing observation in real time, and the method comprises the following steps: selecting a real road network to be subjected to traffic signal control, and taking the real road network as a subsequent training and testing scene; constructing a traffic scene as a training environment of a simulation stage; designing a DRL signal controller by using an MAT encoder-decoder architecture, and training the DRL signal controller in the SUMO traffic simulator to obtain an MAT controller; building a real-time observation completion module based on a feature propagation structure FP; and combining the trained MAT controller and the constructed real-time observation completion module FP to form an FP-MAT combined framework, and putting the FP-MAT combined framework into a corresponding road network to execute a signal control task. According to the method, missing observation in a global state can be automatically identified, real-time and accurate complementation can be carried out, normal work of the signal controller is ensured, and the method has environmental adaptability and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of network traffic control, and in particular relates to a highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time. Background Art

[0002] Network traffic signal control (NTSC) has been rapidly developed around the world as a technology that can most directly improve the overall traffic conditions in a region. Deep reinforcement learning (DRL) has been widely used in the NTSC field in recent years due to its outstanding performance in handling collaborative control tasks in complex environments. In a typical DRL method, each intersection in the road network is regarded as an intelligent agent, which learns the optimal strategy through interaction with the traffic environment. The intelligent agents cooperate in a certain mode to complete the NTSC task, with the goal of significantly improving global traffic efficiency.

[0003] Due to the high cost of training in real-world environments, DRL agents are usually trained in virtual environments created by traffic simulation software and then deployed to real-world road networks to perform signal control tasks. Although DRL methods have been proven to be more effective than traditional methods, they are not currently widely used in the real world. On the one hand, it is difficult to adapt to dynamically changing traffic conditions. Due to algorithm performance limitations, most DRL methods are difficult to converge to the ideal control effect in training scenarios with complex and changing traffic conditions. Therefore, they are usually trained in a single traffic flow mode, which may cause the controller to face overfitting problems and fail to show good adaptability under significant traffic flow changes. On the other hand, unpredictable environmental interference, especially the lack of agent observation due to traffic sensor failure, poses a serious hidden danger to the reliability of DRL methods, which may lead to serious traffic chaos or even system collapse.

[0004] Therefore, using innovative and efficient algorithmic mechanisms to promote DRL agents to learn rich control strategies and converge to ideal performance in complex training environments, and configuring real-time observation completion functions to reduce environmental uncertainty interference, is of great significance for promoting DRL methods from simulation to reality and achieving accurate and adaptive traffic signal control. Summary of the invention

[0005] In order to make up for the shortcomings of the prior art, the present invention aims to provide a highly adaptive road network-level traffic signal control method that can complete missing observations in real time, so as to solve the problems of incomplete observations caused by the difficulty in adapting to dynamically changing traffic conditions and random missing of traffic sensor data. Firstly, a highly adaptive DRL algorithm is constructed based on the Multi-Agent Transformer (MAT) architecture, and the advantages of the agent sequence update mechanism and the high efficiency of the Transformer network are used to reduce the complexity of the multi-agent joint control problem, so as to ensure that the DRL agent converges to the direction of performance improvement in challenging road network scenarios and learns rich and optimal control strategies; then, a real-time observation completion module is constructed with the feature reconstruction algorithm FP based on the diffusion equation, and it is used in combination with a fully trained signal control model in the application stage to eliminate the hidden dangers of uncertain observation missing to DRL and enhance the robustness and reliability of the method.

[0006] To achieve the above purpose, the specific technical solution of the present invention is as follows:

[0007] The highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time comprises the following steps:

[0008] S1. Select a real road network for traffic signal control, convert the original road network file into a SUMO road network file, or directly build the road network in the SUMO road network editor as a subsequent training and testing scenario;

[0009] S2. Based on the road network structure obtained in step S1, a traffic scenario is constructed as a training environment for the simulation stage;

[0010] S3. Based on the training environment constructed in step S2, a DRL signal controller is designed using the MAT encoder-decoder architecture, and the DRL signal controller is trained in the SUMO traffic simulator to obtain a MAT controller;

[0011] S4. Construct real-time observation completion module FP based on feature propagation FP;

[0012] S5. Combine the MAT controller trained in step S3 and the real-time observation completion module FP constructed in step S4 to form a FP-MAT joint framework, which is put into the corresponding road network to perform signal control tasks.

[0013] Furthermore, the specific contents of step S3 include:

[0014] S31. At each time step t, the MAT encoder takes a series of observations of the agent As input, the intersection state observed by agent i at time step t is calculated as: Among them, ln is the total number of import lanes at the intersection where agent i is located, are respectively the vehicle queue length, average vehicle speed and road occupancy rate measured by the lane-level traffic sensor at the import lane l at time step t;

[0015] S32. Design the corresponding action of the intelligent agent based on the optional signal phase of the intersection, that is, t Encoded into a series of latent representations, which are then passed to a decoder to generate the optimal actions for each agent, which are then executed by each agent;

[0016] S33. Based on the impedance index, the reward function is designed. The instantaneous reward of agent i at time step t is calculated as follows: Among them, σ is used to measure the influence of vehicle speed on reward;

[0017] S34. Record the information of a series of intelligent agents at each time step in the simulation process, which is used to update the neural network parameters once after each round. The Bellman loss function and PPO-Clip loss function are used to train the encoder and decoder respectively.

[0018] Furthermore, the specific process of step S4 includes:

[0019] First, according to the road network structure obtained in step S1, the traffic sensor network is constructed as an undirected graph G = (V, E) composed of lane-level traffic sensors, and the adjacency matrix A∈{0,1} is determined by the adjacency relationship between lanes. n×n , and calculate its diffusion matrix Each node in the node set V represents a lane-level traffic sensor in the road network, n is the number of nodes in the node set V, E is the set of edges representing the associations between nodes, and D is the diagonal matrix of A;

[0020] Then, a real-time observation completion module is built based on the FP algorithm. At each time step t, The global state of the road network x is determined by real-time measurements of a series of sensors t ∈R n×d , d is the feature dimension, x t and Iterate and complete the input and output the result Transformed into a series of complete observations of each agent's corresponding intersection.

[0021] Furthermore, in step S5, at each control time step t, the real-time observation completion module uses the global state x of the road network t and diffusion matrix As input, it determines whether there is any missing observation and completes it, and then distributes its output to each intersection agent as input, and outputs the optimal signal phase selection in the current state in real time; at a certain control interval, the controller calculates the current optimal phase and determines whether to execute it; by setting the maximum signal phase duration G max And the minimum signal phase duration G min , avoiding traffic chaos caused by frequent switching of signal phases by the controller.

[0022] Furthermore, the evaluation index of the traffic signal control method is a traffic performance index that can be collected and calculated for the entire road network during the control process.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] (1) This method combines the DRL method of observation completion technology to improve the robustness and efficiency of road network-level traffic signal control under uncertain scenarios with dynamic changes in traffic flow and random missing sensor data. It can not only adapt to various traffic scenarios with complex network structures, dense and time-varying traffic flow, and make adaptive control strategy adjustments to the dynamic changes of traffic flow, but also automatically identify missing observations in the global state and complete them in real time and accurately to ensure the normal operation of the signal controller. It has environmental adaptability and efficiency, great application potential, and strong scalability.

[0025] (2) A highly adaptable DRL architecture, MAT, is used to complete the NTSC task. The sequential update mechanism of the intelligent agents and the high efficiency of the Transformer network ensure the performance advantages of MAT: it can not only accurately extract the high-level associations between intelligent agents at intersections and enhance the collaboration of intelligent agents; it also gives MAT a monotonic performance improvement guarantee by executing the multi-agent advantage decomposition theorem, so that it can smoothly converge to the optimal joint strategy when training in complex traffic environments, thereby improving the global control effect.

[0026] (3) The FP-MAT framework was designed, and a preprocessing module for real-time observation completion was constructed using FP, which effectively avoided the impact of uncertain observability impairment on the reliability of the DRL traffic signal controller. The joint framework can normally complete the NTSC task under harsh traffic conditions where the traffic flow has significant time-variability and the agent observation is seriously missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart for implementing the highly adaptable road network-level traffic signal control method combined with real-time observation and completion of the present invention;

[0028] Figure 2 This is a schematic diagram of the application of a highly adaptable road network-level traffic signal controller combined with real-time observation and completion according to the present invention;

[0029] Figure 3 A diagram showing the road network structure and training vehicle flow settings in an embodiment of the present invention;

[0030] Figure 4 Graph showing the performance test results of the controller under different observation missing rates in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following further describes a highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time according to the present invention in conjunction with the accompanying drawings.

[0032] This embodiment performs traffic signal control on a 4×4 road network in Binjiang District, Hangzhou City, Zhejiang Province. The embodiment includes 16 intersections with traffic lights. According to the road network structure, a set of dense and time-varying traffic flows is designed and constructed as a SUMO traffic flow file for simulation training and testing. The operation parameters and variables involved in this method are shown in Table 1 and Table 2:

[0033] Table 1 (Operation parameters of a traffic signal controller for a road network):

[0034]

[0035]

[0036] Table 2 (a road network traffic signal control variable table):

[0037] Variable Name Variable units Simulation time s Agent observation missing rate ％ Vehicle speed m / s Vehicle delay time s Number of stops freq Congestion Index Vehicle queue length veh

[0038] like Figure 1 and Figure 2 As shown in FIG. 1 , a highly adaptive road network-level traffic signal control method that can complement missing observations in real time is implemented in the following specific steps:

[0039] (1) Obtain the road network file of this embodiment on the OpenStreetMap platform, and convert the original road network file into a SUMO road network file, including road structure data, traffic light data, lane detector data, etc., as a road network scenario for subsequent training and testing.

[0040] (2) Based on the road network structure obtained in step 1, a set of dense and time-varying traffic flows is designed for this embodiment to simulate the random and complex traffic conditions in the real world and construct a challenging traffic scenario, such as Figure 3 As shown in the figure, in the 3600-second simulation process, 0-3200 seconds is the traffic flow input stage. In this stage, we set four different traffic flow conditions with significantly different distributions, aiming to test the robustness and optimality of the DRL algorithm.

[0041] (3) In the simulation training phase, the DRL signal controller is trained in the SUMO traffic simulator based on the training environment constructed in step 2. The DRL-based signal control is modeled as a Markov decision process (MDP), which makes scientific and effective definitions of the observation, action, and reward of the agent.

[0042] At each time step t, the MAT encoder takes a series of observations of the agent As input, the intersection state observed by agent i at time step t is calculated as: (l n is the total number of import lanes at the intersection where agent i is located, are respectively the vehicle queue length, average vehicle speed and road occupancy rate measured by the lane-level traffic sensor at the import lane l at time step t, and o t It is encoded into a series of potential representations, which are then passed to the decoder to generate the optimal action for each agent and executed by each agent. That is, the action of the corresponding agent is designed based on the optional signal phase of the intersection; the reward function is designed based on the impedance index to ensure that the algorithm converges to improve the comprehensive traffic indicators. The immediate reward obtained by agent i after performing the action (time step t) is calculated as follows: Among them, σ is used to measure the influence of vehicle speed on rewards; a series of observations, actions, rewards and other information of the agent are recorded at each time step in the simulation process, which is used to update the neural network parameters once after each round. The encoder and decoder are trained using the Bellman loss function and the PPO-Clip loss function respectively. For example, 200 rounds are trained in the simulation environment, each round lasts 3600s, and the control interval is 5s.

[0043] (4) Build a real-time observation completion module.

[0044] First, based on the road network structure obtained in step 1, an undirected graph G = (V, E) consisting of lane-level traffic sensors is constructed. Each node in the node set V represents a lane-level traffic sensor in the road network. The adjacency matrix A∈{0,1} is determined by the adjacency relationship between lanes. n×n , and calculate its diffusion matrix (D is the diagonal matrix of A), n is the number of nodes in V (i.e., the number of lane-level traffic sensors in the road network), E is the set of edges representing the associations between nodes, and D is the diagonal matrix of A.

[0045] Then, a real-time observation completion module is built based on the FP algorithm. At each time step t, The global state of the road network x is determined by real-time measurements of a series of sensors t ∈R n×d, d is the feature dimension (i.e., the number of traffic indicators measured by the sensor), x t and Iterate and complete the input quickly and accurately, reconstruct the global state with missing observations, and output the complete global state after completion And the output result The FP module is used in combination with the DRL module in the application phase.

[0046] (5) In the application phase, the MAT controller trained in step 3 and the real-time observation completion module FP constructed in step 4 are combined to form Figure 2 The FP-MAT joint framework shown is put into the corresponding road network to perform signal control tasks.

[0047] At each control time step t, the real-time observation completion module takes the global state of the road network x t and diffusion matrix As input, it determines whether there are missing observations and completes them, and then distributes its output to each intersection agent (signal controller) as input, and outputs the optimal signal phase selection in real time based on this. At certain control intervals, the controller calculates the current optimal phase and determines whether to execute it. By setting the maximum signal phase duration G max And the minimum signal phase duration G min , avoiding traffic chaos that may be caused by the controller frequently switching signal phases. The evaluation index of the traffic signal control method is the average vehicle speed of the entire road network during the control process, the average vehicle travel time, the average number of intersection stops or other traffic performance indicators that can be collected and calculated.

[0048] In order to show the tolerance of FP-MAT under high sensor failure rate (i.e. high observation missing rate), the change of vehicle delay in the whole process is used as the evaluation index for testing. In this embodiment, the test is carried out at missing rates of 0, 0.2, 0.3, 0.4, and 0.5, and 10 rounds are tested at each missing rate. In each round, a certain number of sensors are randomly selected and their measurement data are masked to simulate the traffic environment of sensor failure. Figure 4 The performance test results are shown in the figure. The solid line represents the average value of all test rounds, and the shaded part represents the standard deviation. The results show that even in the extreme cases of dense, time-varying traffic flow and severe observation loss, the method can maintain stable and ideal traffic efficiency, showing excellent robustness and effectiveness, proving its potential for application in actual NTSC tasks.

[0049] In summary, this method mainly includes obtaining the road network structure and constructing the traffic flow, calculating the diffusion matrix, training the DRL module in the simulation environment, constructing the real-time observation completion module, forming the final framework, and applying it in real scenarios. Figure 4 Shown is a performance test result diagram of the controller in an embodiment of the present invention under different observation missing rates. The test results show that the invention can achieve stable and excellent performance under extreme conditions of dense, time-varying traffic flow and severe observation missing, proving its adaptability, reliability and efficiency, and showing its application potential and value.

[0050] This method is based on the structural data of the target road network and artificially constructed dense, time-varying traffic flow training data. Deep reinforcement learning training is performed on the target road network in a simulation environment. After sufficient training, the controller model learns rich and optimal control strategies to ensure that the controller model exhibits good adaptability and makes effective decisions under any traffic conditions. It is finally deployed in the corresponding real-world road network scenario, combined with the real-time observation completion module, to ensure that all intersection signal agents receive complete and accurate intersection observations, and adaptively complete signal control tasks in dynamic traffic environments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A highly adaptive network-level traffic signal control method that can complement missing observations in real time. It is characterized in that The following steps are involved: S1. Select a real road network for traffic signal control, convert the original road network file into a SUMO road network file, or directly build the road network in the SUMO road network editor as a subsequent training and testing scenario; S2. Based on the road network structure obtained in step S1, a traffic scenario is constructed as a training environment for the simulation stage; S3. Based on the training environment constructed in step S2, a DRL signal controller is designed using the MAT encoder-decoder architecture, and the DRL signal controller is trained in the SUMO traffic simulator to obtain a MAT controller; S4. Construct real-time observation completion module FP based on feature propagation FP; S5. Combine the MAT controller trained in step S3 and the real-time observation completion module FP constructed in step S4 to form a FP-MAT joint framework, which is put into the corresponding road network to perform signal control tasks.

2. A highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time according to claim 1, It is characterized in that The specific contents of step S3 include: S31. At each time step t, the MAT encoder takes a series of observations of the agent As input, the intersection state observed by agent i at time step t is calculated as: Among them, l n is the total number of import lanes at the intersection where agent i is located, are respectively the vehicle queue length, average vehicle speed and road occupancy rate measured by the lane-level traffic sensor at the import lane l at time step t; S32. Design the corresponding action of the intelligent agent based on the optional signal phase of the intersection, that is, t Encoded into a series of latent representations, which are then passed to a decoder to generate the optimal actions for each agent, which are then executed by each agent; S33. Based on the impedance index, the reward function is designed. The instantaneous reward of agent i at time step t is calculated as follows: Among them, σ is used to measure the influence of vehicle speed on reward; S34. Record the information of a series of intelligent agents at each time step in the simulation process, which is used to update the neural network parameters once after each round. The Bellman loss function and PPO-Clip loss function are used to train the encoder and decoder respectively.

3. A highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time according to claim 2, It is characterized in that The specific process of step S4 includes: S41. According to the road network structure obtained in step S1, the traffic sensor network is constructed as an undirected graph G = (V, E) composed of lane-level traffic sensors, and the adjacency matrix A∈{0,1} is determined by the adjacency relationship between lanes. n×n , and calculate its diffusion matrix Each node in the node set V represents a lane-level traffic sensor in the road network, n is the number of nodes in the node set V, E is the set of edges representing the associations between nodes, and D is the diagonal matrix of A; S42. Based on the FP algorithm, a real-time observation completion module is constructed. At each time step t, The global state of the road network x is determined by real-time measurements of a series of sensors t ∈R n×d , d is the feature dimension, x t and Iterate and complete the input and output the result Transformed into a series of complete observations of each agent's corresponding intersection.

4. A highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time according to claim 3, It is characterized in that In step S5, at each control time step t, the real-time observation completion module uses the global state x of the road network t and diffusion matrix As input, it determines whether there are missing observations and completes them, and then distributes its output to each intersection agent as input, and outputs the optimal signal phase selection in the current state in real time; at certain control intervals, the controller calculates the current optimal phase and determines whether to execute it.

5. A highly adaptable road network-level traffic signal control method capable of supplementing missing observations in real time according to claim 4, It is characterized in that The evaluation index of the traffic signal control method is a traffic performance index that can be collected and calculated for the entire road network during the control process.

Images