Multi-intersection signal cooperative control method based on deep reinforcement learning

Abstract

The invention discloses a multi-intersection signal cooperative control method based on deep reinforcement learning, and the method comprises the steps: firstly constructing a comprehensive reward function which comprises waiting time, queuing length and the influence of adjacent intersections, and constructing a mathematical model of a multi-intersection signal control system based on the reward function, so as to achieve the maximization of a comprehensive reward; and secondly, designing a deep Q network cooperative control algorithm for explicit strategy adjustment, and optimizing and adjusting adjustable strategy function parameters by introducing an explicit strategy adjustment function and utilizing a stochastic gradient descent method. And finally, dynamically adjusting the signal control strategy of each intersection in the road network in a complex and changeable traffic environment by designing a parameter sharing mechanism, thereby realizing collaborative optimization control of multiple intersections. According to the invention, the cooperative control efficiency of a multi-intersection signal system is obviously improved, traffic delay is reduced, and the traffic capacity of a road network is improved.

Description

Technical Field

[0001] The present invention belongs to the field of traffic engineering and traffic information and control systems, and relates to the technical field of multi-intersection signal collaborative control, and more specifically, to a multi-intersection signal collaborative control method based on deep reinforcement learning. Background Art

[0002] Most of the research on traffic signal timing optimization is based on the algorithm of optimization model. With the development of intelligent transportation technology, the multi-agent based deep reinforcement learning (DRL) method has attracted widespread attention in the field of traffic signal control. By treating each intersection as an independent agent, using feedback learning and information sharing, signal linkage control is realized, which enhances the overall operation efficiency of the transportation system. In the multi-intersection signal coordination control framework based on deep reinforcement learning, the agent usually represents the traffic lights, and the environment corresponds to the intersections with traffic lights. Each agent makes decisions based on the traffic status (such as vehicle position, speed, queue length, etc.), aiming to optimize signal control by maximizing the overall reward (such as reducing vehicle waiting time). Many domestic and foreign studies have shown that the use of deep reinforcement learning technology can effectively improve the coordination and adaptability of traffic signal control. For example, Nishi et al. extracted the geometric features of the road network through a graph convolutional neural network and combined the neural fitting Q iteration algorithm to realize distributed signal control (IEEE, 2018: 877-883); Chu et al. proposed a multi-agent A2C algorithm to optimize the signal control of adjacent intersections through a mixed reward function (IEEE Transactions on Intelligent Transportation Systems, 2019, 21(3): 1086-1095); HUO et al. used an end-to-end learning model in a V2X environment to achieve efficient intersection coordination control. Although these methods have achieved certain results, most of them are aimed at single or adjacent intersections. The signal control of multiple intersections still faces problems such as high model complexity and unstable control strategy. To this end, the present invention proposes a multi-intersection signal collaborative control method based on deep reinforcement learning. Through explicit strategy adjustment functions and parameter sharing mechanisms, the interpretability of strategy selection is improved, and the complexity and noise impact of the algorithm are reduced. The algorithm uses queue length and waiting time as state variables, signal phase as action space, and incorporates the influence of adjacent intersections into the reward function. The introduction of this technology provides a new solution for traffic signal timing optimization, which is particularly suitable for multi-intersection collaborative control scenarios with complex traffic flow during peak hours. Summary of the invention

[0003] Technical problem: In view of the shortcomings of the existing multi-intersection signal control based on deep reinforcement learning, such as high model complexity and weak interpretability of strategy selection, the purpose of the present invention is to provide a multi-intersection signal collaborative control method based on deep reinforcement learning. By introducing explicit strategy adjustment functions and parameter sharing mechanisms, the interpretability of algorithm strategy selection is improved and its complexity and noise impact are reduced, thereby realizing multi-intersection signal collaborative control, reducing the average queue length and average waiting time, and improving traffic efficiency. Technical solution: To solve the above technical problems, the multi-intersection signal coordinated control method based on deep reinforcement learning of the present invention comprises the following steps:

[0004] Step 1: Construct a comprehensive reward function that includes waiting time, queue length, and the impact of adjacent intersections, and build a mathematical model with the goal of maximizing the comprehensive reward function of the multi-intersection signal control system; Step 2: By introducing an explicit policy adjustment function, an explicit policy adjustment deep Q network collaborative control algorithm is designed; Step 3: Use the stochastic gradient descent method to adjust the parameters of the adjustable policy function; Step 4: By designing a parameter sharing mechanism, the signal control strategy of each intersection in the road network is dynamically adjusted to achieve coordinated optimization control of multiple intersections. The comprehensive reward function calculation method in step 1 includes the following steps: Step 11: The goal is to maximize the sum of the reward functions of the multi-intersection traffic signal control system, as shown in formula (1): Where: represents the comprehensive reward function of each traffic light, S represents the set of traffic lights, where each traffic light s∈S; The comprehensive reward function for each traffic light The negative reward R of the waiting time s , Negative rewards for queue length and the impact of adjacent intersections The sum of the three parts is shown in formula (2): Among them, the negative reward R of waiting time s Calculated by formula (3): Where: L i represents the set of lanes at intersection i, w(l) represents the waiting time of each lane l; Negative rewards for queue length Calculated by formula (4): Where: traffic movement (l, m) represents the process from lane l of the entrance road to lane m of the exit road, p(l, m) represents the pressure of lane-level traffic movement, and p(l, m) is calculated by formula (5): In the formula, represents the vehicle density of a lane, where n(l) is the number of vehicles on lane l and C(l) is the maximum capacity of lane l; The influence of adjacent intersections Calculated by formula (6): In the formula, QW i represents the weighted sum of the queue length and waiting time at intersection i, calculated by formula (7): Where q(l) is the queue length of each lane l, w(l) is the waiting time of lane l, and L i represents the set of lanes at each intersection i. The design of an explicit strategy to adjust the deep Q network cooperative control algorithm in step 2 includes the following steps: Step 21: Introduce an explicit adjustable strategy function G(s,a;θ), in which s represents the current state, including traffic flow and current traffic light state; a represents the optional action to change the traffic light state, and θ is the parameter of the adjustable function; by optimizing the parameter θ of the strategy function G, the optimal action selection based on the Q function under a given state is determined; for a given state s and action a, when the action is the optimal choice of the Q function, the target value is 1; otherwise, the target value is 0. In step 3, the parameter θ of the adjustable policy function is adjusted using the stochastic gradient descent method, which includes the following steps: Step 31: Use the softmax function to convert the output of the policy function G(s, a; θ) into a probability distribution, so that the selection probability of each possible action is proportional to the corresponding Q value, as shown in formula (8): Where: π(a|s; θ) represents the probability of selecting action a in state s; Step 32: The loss function is used to measure the gap between the Q value output by the strategy function G and the target Q value. The Huber loss function L(θ) is shown in formula (9): Where: Q(s i ,a i ; θ) is the predicted Q value; N represents the number of samples, the size of the data batch used to calculate the loss; yi is the target Q value, usually calculated by Bellman equation (10): y i =r i +γmax a′ Q(s i+1 ,a′;θ - ) (10) Where: r i is the immediate reward, γ discount factor, between 0 and 1, used to measure the weight of future rewards. The closer it is to 1, the higher the importance of future rewards; max a′ Q(s i+1 ,a′;θ - ) means in the next state s i+1 The maximum Q value that can be obtained by selecting action a′ under the optimal strategy represents the maximum expected cumulative reward, θ - are the parameters of the target network, s i+1 is the new state to which the action is transferred after execution; Step 33: Use the gradient of the loss function to update the parameters to minimize the value of the loss function. By calculating the gradient of the loss function, the weights of the network are gradually adjusted so that the value of the loss function gradually decreases, and finally the predicted value output by the network is closer to the target value, as shown in formula (11): Where η is the learning rate, is the gradient of the loss function with respect to the parameter θ; the softmax function converts the output of the neural network into a probability distribution for selecting actions; the loss function measures the gap between the predicted result of the softmax output and the target Q value; the stochastic gradient descent uses the error calculated by the loss function to update the parameters of the softmax layer of the neural network through the gradient descent method. The parameter sharing mechanism design method in step 4 includes the following steps: Step 41: First, the intelligent agent at each intersection independently learns and makes decisions based on the local status of vehicle flow, queue length, and phase duration, and quickly adapts to the local traffic environment in which it is located; second, by evaluating the difference in upstream and downstream queue lengths between adjacent intersections, the signal timing parameters are dynamically adjusted to ensure the balance of traffic flow between intersections and reduce vehicle queues and congestion between intersections; by designing a shared strategy network and global hyperparameters, the model complexity of individual intelligent agents is reduced, repeated calculations are avoided, and training time is shortened; finally, the intelligent agents at each intersection share the experience replay buffer, learn from each other's signal phase selection experience at different intersections during the training process, and use the decision-making experience of other intersections to improve their own strategies, enhance the global optimality of the strategy, and achieve a more efficient traffic signal control strategy.

[0005] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0006] Different from the existing intersection signal cooperative control technology based on deep reinforcement learning, which mainly focuses on factors such as vehicle delay, queue length, travel time, and green wave coordination in the reward function design, and ignores the mutual influence between adjacent intersections in a multi-intersection system, this invention innovatively introduces the mutual influence factors between adjacent intersections when constructing a comprehensive reward function, and combines the dynamic changes of vehicle waiting time and queue length to perform more refined control of complex traffic flow. Secondly, in view of the fact that the policy adjustment function in existing research usually only adopts adjustment methods such as dynamically adjusting the exploration rate (such as ε-greedy strategy), the present invention proposes an explicit policy adjustment mechanism. By designing a deep Q-network (DQN) collaborative control algorithm with explicit policy adjustment, the interpretability and stability of policy optimization are improved. In addition, the present invention designs a parameter sharing mechanism for the coordinated optimization control of multiple intersections, so that each intersection can share key information in a complex and changeable traffic environment and dynamically adjust the signal control strategy. This mechanism effectively reduces the computational complexity in the multi-intersection system, enhances the coordination and robustness of the entire road network system, significantly reduces traffic delays, and improves the overall traffic capacity of the road network. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a flow chart of the method of the present invention;

[0008] Figure 2 A schematic diagram of the research object of the method of the present invention;

[0009] Figure 3 It is a schematic diagram of the multi-intersection parameter sharing mechanism;

[0010] Figure 4 This is the convergence curve of the algorithm’s average reward function. DETAILED DESCRIPTION

[0011] In conjunction with the accompanying drawings and embodiments, the technical solution of the present invention is described in detail as follows:

[0012] Figure 2 The schematic diagram of the research object of the method of the present invention is that the intersection pressure is |2+1+5+2-3-0-2-0|=5. The pressure indicates the imbalance between the density of vehicles entering and leaving. The greater the pressure, the more unbalanced the vehicle distribution. Figure 3The multi-intersection parameter sharing mechanism is demonstrated. First, the system obtains state variables such as the length of the vehicle queue and the waiting time at each intersection from the environment, and passes these state variables as input to the deep reinforcement learning network. These inputs are processed by a multi-layer neural network, and the network outputs the optimized traffic signal control strategy and generates the corresponding signal light switching action. Each intersection performs signal light switching according to the action generated by the network, updates the signal state, and thus affects the traffic flow at the intersection. After each action is executed, the intersection feeds back the new state variables and reward signals to the deep reinforcement learning network, where the reward signal is used to evaluate the effect of the action taken. This feedback loop enables the network to gradually optimize the control strategy by continuously accumulating and utilizing feedback information, thereby achieving traffic flow optimization for the entire road network. In addition, the replay memory in this mechanism is also shared, which means that the agent can reuse previous experience during training, allowing all intersections to share historical experience data of other intersections. This design not only improves computational efficiency, but also significantly enhances the generalization ability of the model, enabling it to effectively coordinate between different intersections, thereby adapting to more complex traffic environments and improving the overall traffic capacity of the road network.

[0013] A 2×2 road network was constructed as the research area. The network contains four traffic signal-controlled intersections. The internal structure of all intersections is the same, and each road is bidirectional. Each intersection is set with signal phases in four directions, namely east-west straight, east-west left turn, north-south straight, and north-south left turn. The distance between intersections is 500 meters, and the distance between the end of the lane and the stop line of the intersection is 250 meters. A random variable traffic flow model is used to simulate the flow fluctuations of ordinary social vehicles during peak hours. The minimum green light time of each intersection is 15s, and the maximum green light time is 60s. A total of 300 iterations are performed, each iteration contains 400 simulation steps, and the simulation time of each step is 10 seconds. The yellow light duration used in the simulation process is 3 seconds.

[0014] The specific parameters of the model are set as follows: First, in order to limit the range of consideration when selecting the optimal action, the maximum distance is set to 200 meters. The number of batch samples used in each training is 32 to ensure that the experience samples can be fully utilized during the training process. In order to measure the importance of future rewards, the discount factor is set to 0.99, and the model pays great attention to future rewards when making decisions. In addition, the learning rate is set to 0.75. During the exploration process, the initial exploration rate is set to 1.0, that is, at the beginning of training, the agent selects actions completely randomly, and as the training progresses, the exploration rate gradually decreases, and the final exploration rate is set to 0.0, indicating that the model will completely rely on the learned strategy to select the optimal action. In order to control the speed of change of the exploration rate from the initial value to the final value, the decay value of the exploration rate is set to 220. The update frequency of the target network is set to every 500 steps.

[0015] Under the premise of the same simulation environment and algorithm parameters, 10 simulation experiments were performed synchronously. The reward value of each training round was recorded in each simulation experiment, and the data of the 10 experiments were averaged. The experimental results obtained are as follows: Figure 4 As shown. Figure 4 It can be seen that in the early stage of training, about the first 50 rounds, the reward is low and fluctuates greatly because the model is still exploring the environment. After 50 rounds, the average reward generally shows a relatively stable and gradually rising trend, which indicates that the model is gradually learning and optimizing the strategy through training with a large number of experience samples. After about 200 rounds, the reward curve tends to be stable, the fluctuation amplitude decreases and remains at a high level, indicating that the model is close to convergence, that is, it has found a relatively stable and better strategy. In the last 50 rounds, the average reward basically remains at a fixed level, indicating that the performance of the model has stabilized, that is, it has basically converged under the current training environment and strategy.

[0016] The model establishment takes into account the two optimization objectives of queue length and waiting time. Therefore, the maximum queue length, average queue length, and average waiting time are selected as evaluation indicators to evaluate the control effect. At the same time, 6 comparison algorithms are selected as control experiments. The evaluation indicators output by each algorithm are tested under the same environment to evaluate the effectiveness of the model. The 6 comparison methods are: (1) DQN-based signal collaborative control algorithm (comparison method 1): This algorithm approximates the Q-value function through a deep neural network, and the agent uses the ε-greedy strategy to select the phase of the traffic light. In the multi-agent collaborative mode, each agent collaboratively optimizes the signal control through information sharing. (2) Signal-independent control algorithm based on DQN (comparative method 2): This algorithm also approximates the Q-value function through a deep neural network and uses the ε-greedy strategy to optimize signal control. In the independent control mode, each agent makes signal control decisions independently based on the local state without relying on the information of other agents. (3) Random action selection control algorithm (comparative method 3): randomly selects the phase change of traffic lights to control traffic flow without considering the current state or historical data. (4) Maximum pressure signal control algorithm (comparison method 4): By calculating the phase pressure (the difference between the number of waiting vehicles and the flow demand of the adjacent phase), the maximum pressure phase is selected to switch the green light, reducing waiting time and congestion. (5) Maximum wavefront pressure signal control algorithm (comparative method 5): estimate traffic wave propagation, calculate wavefront pressure, select the maximum pressure phase to switch the green light, and reduce congestion and delays. (6) Timing control algorithm (comparative method 6): The traffic lights at multiple intersections are synchronized through a preset timetable to form a green wave band, optimize the flow, and reduce waiting time. Table 1 shows the performance of different control algorithms and their optimization percentage relative to the timing control algorithm.

[0017] Table 1 Performance comparison of different control algorithms It can be seen from the above table that in terms of overall performance, the method of the present invention performs best in terms of the three indicators of maximum queue length, average queue length and average waiting time. The present invention is not limited to this single embodiment, and any other changes, modifications, substitutions, combinations, simplifications that deviate from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A multi-intersection signal coordinated control method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Construct a comprehensive reward function that includes waiting time, queue length, and the impact of adjacent intersections, and build a mathematical model with the goal of maximizing the comprehensive reward function of the multi-intersection signal control system; Step 2: By introducing an explicit policy adjustment function, an explicit policy adjustment deep Q network collaborative control algorithm is designed; Step 3: Use the stochastic gradient descent method to adjust the parameters of the adjustable policy function; Step 4: By designing a parameter sharing mechanism, the signal control strategy of each intersection in the road network is dynamically adjusted to achieve coordinated optimization control of multiple intersections.

2. According to the method of multi-intersection signal coordinated control based on deep reinforcement learning in claim 1, it is characterized in that: The comprehensive reward function calculation method in step 1 comprises the following steps: Step 11: The goal is to maximize the sum of the reward functions of the multi-intersection traffic signal control system, as shown in formula (1): Where: represents the comprehensive reward function of each traffic light, S represents the set of traffic lights, where each traffic light s∈S; The comprehensive reward function for each traffic light The negative reward R of the waiting time s , Negative rewards for queue length and the impact of adjacent intersections The sum of the three parts is shown in formula (2): Among them, the negative reward R of waiting time s Calculated by formula (3): Where: L i represents the set of lanes at intersection i, w(l) represents the waiting time of each lane l; Negative rewards for queue length Calculated by formula (4): Where: traffic movement (l,m) represents the process from lane l of the entrance road to lane m of the exit road, p(;,m) represents the pressure of lane-level traffic movement, and p(l,m) is calculated by formula (5): In the formula, represents the vehicle density of a lane, where n(l) is the number of vehicles on lane l and C(l) is the maximum capacity of lane l; The influence of adjacent intersections Calculated by formula (6): In the formula, QW i represents the weighted sum of the queue length and waiting time at intersection i, calculated by formula (7): Where q(l) is the queue length of each lane l, w(l) is the waiting time of lane l, and L i represents the set of lanes at each intersection i.

3. The method for coordinated control of multiple intersection signals based on deep reinforcement learning according to claim 1 is characterized in that: The design of the explicit strategy to adjust the deep Q network collaborative control algorithm in step 2 includes the following steps: Step 21: Introduce an explicit adjustable policy function G(s, a; θ), where s represents the current state, including traffic flow and current traffic light state; a represents the optional action to change the traffic light state, and θ is the parameter of the adjustable function; by optimizing the parameter θ of the policy function G, the optimal action selection based on the Q function under a given state is determined; for a given state s and action a, when the action is the optimal choice of the Q function, the target value is 1; otherwise, the target value is 0; The step 3 of adjusting the parameter θ of the adjustable strategy function by using the stochastic gradient descent method comprises the following steps: Step 31: Use the softmax function to convert the output of the policy function G(s, a; γ) into a probability distribution, so that the selection probability of each possible action is proportional to the corresponding Q value, as shown in formula (8): π(a|s; θ) represents the probability of selecting action a in state s; Step 32: The loss function is used to measure the gap between the Q value output by the strategy function G and the target Q value. The Huber loss function L(θ) is shown in formula (9): Among them, Q(s i , a i ; θ) is the predicted Q value; N represents the number of samples, the size of the data batch used to calculate the loss; y i is the target Q value, usually calculated by Bellman equation (10): y i =r i +γmax a′ Q(s i+1 ,a′;θ - ) (10) Among them, r i is the immediate reward, γ discount factor, between 0 and 1, used to measure the weight of future rewards. The closer it is to 1, the higher the importance of future rewards; max a′ Q(s i+1 , a′; θ - ) means in the next state s i+1 The maximum Q value that can be obtained by selecting action a′ under the optimal strategy represents the maximum expected cumulative reward, θ - is the parameter of the target network, s i+1 is the new state to which the action is transferred after execution; Step 33: Use the gradient of the loss function to update the parameters to minimize the value of the loss function. By calculating the gradient of the loss function, the weights of the network are gradually adjusted so that the value of the loss function gradually decreases, and finally the predicted value output by the network is closer to the target value, as shown in formula (11): Where η is the learning rate, is the gradient of the loss function with respect to the parameter θ; the softmax function converts the output of the neural network into a probability distribution for selecting actions; the loss function measures the gap between the predicted result of the softmax output and the target Q value; the stochastic gradient descent uses the error calculated by the loss function to update the parameters of the softmax layer of the neural network through the gradient descent method.

4. The method for coordinated control of multiple intersection signals based on deep reinforcement learning according to claim 1 is characterized in that: The parameter sharing mechanism design method in step 4 comprises the following steps: Step 41: First, the intelligent agent at each intersection independently learns and makes decisions based on the local status of vehicle flow, queue length, and phase duration, and quickly adapts to the local traffic environment in which it is located; second, by evaluating the difference in upstream and downstream queue lengths between adjacent intersections, the signal timing parameters are dynamically adjusted to ensure the balance of traffic flow between intersections and reduce vehicle queues and congestion between intersections; by designing a shared strategy network and global hyperparameters, the model complexity of individual intelligent agents is reduced, repeated calculations are avoided, and training time is shortened; finally, the intelligent agents at each intersection share the experience replay buffer, learn from each other's signal phase selection experience at different intersections during the training process, and use the decision-making experience of other intersections to improve their own strategies, enhance the global optimality of the strategy, and achieve a more efficient traffic signal control strategy.

Images