Traffic signal control method and system combining reinforcement learning and model predictive control

Abstract

The application provides an intelligent adaptive traffic signal control method combining reinforcement learning and model predictive control, and belongs to the technical field of intelligent traffic.The application uses a Q-Learning algorithm to continuously optimize an MFD combination selection strategy through online learning, so that the system can dynamically adapt to different traffic flow modes and changes.The Q-Learning algorithm combines an epsilon-greedy selection strategy, gradually improves the control effect through a reward mechanism, and effectively improves the timing efficiency of the boundary signal light.In the signal optimization of the internal area, the application adopts a Cyclic MaxPressure algorithm to dynamically adjust the signal timing according to the real-time traffic flow distribution, so that the signal timing can quickly respond to the randomness and complexity of the traffic flow, thereby effectively relieving the congestion at the local intersection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to a traffic signal control method and system that combines reinforcement learning and model predictive control. Background Technology

[0002] With rapid urbanization, traffic congestion has become a global problem, urgently requiring effective solutions. Traditional fixed-cycle traffic signal control methods often fail to meet the demands of complex traffic conditions due to their lack of real-time adaptability to dynamic traffic situations. In contrast, adaptive traffic signal control methods, with their flexibility and dynamic response capabilities, are gradually becoming an important research area for alleviating urban traffic congestion. Model Predictive Control (MPC) is a control method based on optimization prediction. By constructing a predictive model of the system, MPC uses current traffic conditions and historical data to predict future traffic flow changes and calculates the optimal control input guided by the optimization objective. Because it can dynamically adjust traffic signal timing within a finite future timeframe, MPC has been widely used in traffic signal control. However, MPC in practical applications highly depends on the accuracy of the predictive model, and due to the nonlinear, random, and time-varying characteristics of traffic flow, relying solely on a fixed model may not achieve the desired control effect. Reinforcement Learning (RL) is an adaptive optimization method based on experimentation and feedback. An agent continuously updates its strategy by interacting with the environment to maximize long-term rewards. Reinforcement Learning does not rely on accurate traffic flow models and shows great potential in complex and dynamic traffic environments. However, traditional reinforcement learning may face problems such as slow convergence speed and unstable control effects during the exploration and learning process, especially in the continuous time and state space of traffic signals, where relying solely on reinforcement learning may be difficult to achieve efficient control. Therefore, a combination of reinforcement learning and model predictive control is used to achieve dynamic adjustment of traffic signals in urban road networks.

[0003] The current problems with MPC strategies are as follows: 1) MPC needs to solve an optimization problem in each control cycle. As the number of control variables, the prediction time domain length, and constraints increases, the optimization problem becomes more complex. This computational requirement limits the application of MPC in systems with high real-time requirements or large-scale systems. 2) MPC requires an accurate prediction model to predict the future dynamics of the system. If the accuracy of the prediction model is not high, the control effect may decrease significantly. 3) When the system has significant nonlinearity or a high-dimensional state space, the performance of MPC will be limited because traditional optimization algorithms may not be able to efficiently solve nonlinear or high-dimensional problems.

[0004] Problems with Reinforcement Learning (RL): 1) RL typically requires a large amount of interactive data during training, especially in continuous state and action space problems like traffic signal control, where sampling efficiency is low, leading to long training times. 2) RL needs to find a balance between exploring the unknown environment and utilizing existing policies. Insufficient exploration may prevent the algorithm from discovering the global optimum, while excessive exploration can lead to unnecessary training overhead. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of low efficiency in road network traffic flow allocation and low accuracy in model prediction in existing traffic signal control methods, and to provide a traffic signal control method and system that combines reinforcement learning and model predictive control.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A traffic signal control method combining reinforcement learning and model predictive control includes the following steps:

[0008] The road network area is divided into external areas. and internal area Collect traffic data from various regions and combine it with the traffic demand of each region of the road network to fit different MFD combinations;

[0009] A mathematical optimization model is established based on different MFD combinations, with the goal of minimizing the number of vehicles in the road network area.

[0010] Based on the established mathematical optimization model, the Q-Learning algorithm is combined with... - The greedy selection strategy selects the MFD combination with the highest Q value, and the boundary traffic light timing is obtained by rolling optimization through the MPC controller;

[0011] Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at intersections in congested areas within each region, and the internal traffic signal timing is obtained.

[0012] The timing of the boundary traffic lights is for the outer area. With internal area Traffic light timings for intersecting lanes;

[0013] The internal traffic light timing is for the external area. With internal area The signal light timings for each lane within their respective areas.

[0014] The section on dividing the road network area into external areas and internal area In the process of collecting traffic data from various regions and fitting different MFD combinations based on the traffic demand of each region of the road network, the formula for MFD is defined as follows:

[0015]

[0016] in, Indicates the area middle The process of completing the flow, higher-order items Used to describe the nonlinear effects during congestion phases, quadratic terms. Used to capture the nonlinear change in traffic capacity with increasing density, linear term This reflects the characteristics of vehicles in free-flowing motion. , , These are parameters to be determined, obtained through polynomial fitting based on historical traffic data from traffic flow simulation.

[0017] In the step of establishing a mathematical optimization model based on different MFD combinations with the objective of minimizing the number of vehicles in the road network area, the established mathematical optimization model is as follows:

[0018]

[0019] in, For the final simulation time, for Both the starting and ending points of the time are in the external region. The cumulative number of vehicles, for The starting point of time is the outer region The endpoint is the internal area. The cumulative number of vehicles, for The starting point is region 2 and the ending point is the inner region. The cumulative number of vehicles, for Both the starting and ending points of the time are within the internal area. The cumulative number of vehicles.

[0020] The established mathematical optimization model is used in conjunction with the Q-Learning algorithm. - In the step of updating the Q-value using a greedy selection strategy, choosing the MFD combination with the highest updated Q-value, and then using the MPC controller for rolling optimization to obtain the boundary traffic light timings, the Q-Learning algorithm is combined with... The specific method for updating the Q value using the greedy selection strategy is as follows:

[0021] Combine the MFDs of the two regions into an action set of the Q table, and initialize the Q value of each action in the action set to 0;

[0022] Based on the initialized action set, Epsilon adopts linear decay and updates the exploration probability through an ε-greedy strategy to obtain the epsilon value at each time step and update the Q table;

[0023] A probability is randomly generated and compared with the epsilon value. If the probability is less than the epsilon value, a random action is explored; otherwise, the action with the largest Q value in the updated Q-table is selected.

[0024] Perform the above action to obtain predicted vehicle counts for two areas, and then calculate the reward by comparing the predicted values ​​with the actual observed values.

[0025] Based on the reward calculation results and the actual observed number of vehicles in the two regions, the Q-values ​​in the Q-table are updated using the Q-learning update rules.

[0026] The established mathematical optimization model is used in conjunction with the Q-Learning algorithm. - In the step of updating the Q-value using a greedy selection strategy and choosing the MFD combination with the highest updated Q-value, the boundary traffic light timing is obtained through rolling optimization using an MPC controller. The specific method for obtaining the boundary traffic light timing through rolling optimization using an MPC controller is as follows:

[0027] Based on the updated Q table, the MFD combination with the highest Q value is selected, and a prediction model is constructed by combining it with real-time traffic conditions.

[0028] Based on the constructed predictive model, future traffic flow in the two regions is predicted;

[0029] Based on the constructed prediction model, the internal region and external areas With the objective of minimizing the cumulative number of vehicles, the boundary control input is calculated using the SLSQP optimization algorithm to obtain the boundary control value;

[0030] Calculate the traffic flow at the boundary transition based on the boundary control values ​​and the traffic flow in the two areas.

[0031] The traffic flow transferred from the boundary is distributed to each lane at the boundary intersection, and the green light time is calculated to obtain the boundary traffic signal timing.

[0032] The formula for calculating the vehicle flow during boundary transition, based on the boundary control value and the traffic flow in both areas, is as follows:

[0033]

[0034] in, yes Time zone Transfer to adjacent sub-region Traffic flow, The density ratio within the region, These are boundary control values.

[0035] The method for distributing the traffic flow transferred at the boundary to each lane of the boundary intersection, calculating the green light time, and obtaining the boundary traffic signal timing is as follows:

[0036] The flow rate for each lane is calculated using the following formula:

[0037]

[0038] Transfer traffic to the boundary The lanes allocated to the boundary intersection are determined according to lane ratio. Obtain the boundary intersection lane The formula for calculating the green light time for traffic diversion is as follows:

[0039]

[0040] in For saturated flow rate, For the signal period time, for Time Lane The green light timing.

[0041] The specific method used in the step of optimizing traffic flow distribution at intersections in congested areas within each region using the Cyclic MaxPressure algorithm based on the collected traffic data to obtain the internal traffic light timing is as follows:

[0042] Define a fixed signal period and the set of signal phases for each intersection within the two regions;

[0043] Based on the signal phase set of each intersection, the MaxPressure algorithm is used to calculate the degree of traffic flow imbalance at each intersection, which is then used as the pressure at each intersection.

[0044] Compare the pressure at each intersection within the same cycle, select the signal phase of the intersection with the highest pressure, and record it as the maximum phase signal;

[0045] The Cyclic MaxPressure algorithm is used to dynamically allocate green light time, prioritizing the allocation of green light time for the largest phase signal, to obtain the traffic signal timing of the intersection;

[0046] Repeat the above steps based on real-time traffic flow to dynamically adjust the green light time.

[0047] The method for dynamically allocating green light time using the Cyclic MaxPressure algorithm, prioritizing the allocation of green light time for the largest phase signal, to obtain the traffic light timing at the intersection is as follows:

[0048] The proportion of green time for each phase is determined by probability allocation. The calculation formula is as follows:

[0049]

[0050] in For phase The proportion of green light time, For phase Pressure weight, This is an adjustment coefficient used to control the sensitivity of the weight's influence. This represents the normalized denominator, ensuring that the sum of all phase proportions is 1;

[0051] phase The green light timing is:

[0052]

[0053] in, The period is the signal period.

[0054] A traffic signal control system combining reinforcement learning and model predictive control includes:

[0055] The MFD fitting module is used to divide the road network area into external regions. and internal area Collect traffic data from various regions and combine it with the traffic demand of each region of the road network to fit different MFD combinations;

[0056] The optimization model building module is used to build a mathematical optimization model based on different MFD combinations, with the goal of minimizing the number of vehicles in the road network area;

[0057] The boundary traffic light timing module is used to combine the Q-Learning algorithm with an established mathematical optimization model. - The greedy selection strategy selects the MFD combination with the highest Q value, and the boundary traffic light timing is obtained by rolling optimization through the MPC controller;

[0058] The internal traffic signal timing module is used to optimize the traffic flow distribution at intersections in congested areas within each region based on collected traffic data and the Cyclic MaxPressure algorithm, thereby obtaining the internal traffic signal timing.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] This invention provides a traffic signal control method and system combining reinforcement learning and model predictive control, comprising the following steps: dividing the road network area into external areas. and internal area Traffic data from various regions is collected, and different MFD combinations are fitted based on the traffic demand of each region of the road network. Based on these different MFD combinations, a mathematical optimization model is established with the objective of minimizing the number of vehicles in each region of the road network. Finally, based on the established mathematical optimization model, the Q-Learning algorithm is used in conjunction with... The greedy selection strategy chooses the MFD combination with the highest Q-value, and the solution is obtained through rolling optimization by the MPC controller to obtain the boundary traffic light timing. Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at intersections in congested areas within each region, obtaining the internal traffic light timing. By dividing the traffic network into regions and applying different optimization strategies to the boundary and internal regions, zonal optimization is achieved, thereby significantly improving the traffic flow distribution efficiency of the entire network. In terms of boundary traffic light control, this invention utilizes the Q-Learning algorithm to continuously optimize the MFD combination selection strategy through online learning, enabling the system to dynamically adapt to different traffic flow patterns and changes. In terms of signal optimization in the internal regions, this invention uses the Cyclic MaxPressure algorithm to dynamically adjust the traffic light timing according to the real-time traffic flow distribution, enabling it to quickly respond to the randomness and complexity of traffic flow, improving the accuracy of model prediction, and thus effectively alleviating congestion at local intersections.

[0061] Furthermore, the Q-Learning algorithm combines... - The greedy selection strategy, through a reward mechanism, gradually improves the control effect and effectively enhances the timing efficiency of boundary traffic lights.

[0062] Furthermore, by alleviating localized congestion and preventing its spread to other areas, this invention effectively reduces the total number of vehicles in the traffic network and improves the overall traffic capacity of the road network. Attached Figure Description

[0063] Figure 1 This is a flowchart of the present invention;

[0064] Figure 2This is a schematic diagram of a two-region road network based on SUMO simulation, as presented in this invention.

[0065] Figure 3 This is a macroscopic basic diagram of the present invention;

[0066] Figure 4 This is a schematic diagram illustrating the optimized control method of the present invention, which combines Q-Learning and MPC controllers for use in traffic simulation. Detailed Implementation

[0067] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0068] Example 1

[0069] like Figure 1 As shown, a traffic signal control method combining reinforcement learning and model predictive control includes the following steps:

[0070] S1: Divide the road network area into external areas and internal area Collect traffic data from various regions and combine it with the traffic demand of each region of the road network to fit different MFD combinations;

[0071] S2: Based on different MFD combinations, establish a mathematical optimization model with the goal of minimizing the number of vehicles in the road network area;

[0072] S3: Based on the established mathematical optimization model, the Q-Learning algorithm is combined with... - The greedy selection strategy selects the MFD combination with the highest Q value, and the boundary traffic light timing is obtained by rolling optimization through the MPC controller;

[0073] S4: Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at intersections in congested areas within each region, and the internal traffic signal timing is obtained.

[0074] The timing of the boundary traffic lights is for the outer area. With internal area Traffic light timings for intersecting lanes;

[0075] The internal traffic light timing is for the external area. With internal area The signal light timings for each lane within their respective areas.

[0076] Example 2

[0077] In S1, such as Figure 2As shown, the road network area is divided into zones, and basic parameters such as the length of each road segment and the number of lanes are set according to the actual situation. In this study, the road network is divided into two homogeneous regions with different characteristics, forming a heterogeneous traffic network. One of these regions is the external region. (Area 1), the other is an internal area. (Area 2). The road network is divided into two sub-areas with an inclusion relationship, and traffic flow between these two sub-areas is mainly exchanged through 12 intersections. The two areas... Each region has a good MFD (Mean Traffic Demand), and for a dual-region system, there are two endogenous traffic demands. In the middle, use (veh / s) indicates that in In the middle, use (veh / s) means, and The generated destination is and Two exogenous traffic demands (i.e., border traffic) are respectively used as and (veh / s) represents this. Therefore, there are four different state variables. (veh), of which It means that in Time zone Arrive at the destination The total number of vehicles, the same (veh) indicates in Time zone The total cumulative number of vehicles in the system.

[0078] The macro fundamental chart MFD is used (veh / s) indicates that it represents the region. middle The process of completing the journey, such as Figure 2 As shown, the shape of MFD is approximately a right-sloping, asymmetric unimodal curve, therefore it can be used... The function is defined as a third-order function, and formula (1) is as follows:

[0079]

[0080] Among them, higher-order terms Used to describe the trend of decreased traffic capacity at high density, i.e., the nonlinear effect during congestion, quadratic term. Used to capture the nonlinear change in traffic capacity with increasing density, suitable for medium density ranges. Linear term This reflects the direct impact of vehicle numbers on traffic capacity at low density, i.e., the characteristics of vehicles flowing freely. , , These are parameters to be determined, obtained through polynomial fitting based on historical traffic data from traffic flow simulation, and used to describe some characteristics of different areas. For example... Figure 3 As shown, the horizontal axis (cumulative number of vehicles in the road network) represents the total number of vehicles on all roads in the region, which is the overall quantitative indicator of traffic density, usually expressed in terms of vehicle count (veh). This represents the critical cumulative number of vehicles, corresponding to the maximum traffic capacity. The vertical axis represents the cumulative number of vehicles in congestion, indicating the total number of vehicles when completely congested. (Vertical axis: Travel vehicle completion rate)

[0081] It represents the number of vehicles that can pass through a region per unit time (veh / s), reflecting the traffic capacity of the road network. MFD typically consists of a peak and two decline phases, corresponding to three states: free flow, critical flow, and congested flow. The free flow phase ( <= In a road network with relatively few vehicles and minimal interference between them, traffic capacity increases with the number of vehicles, showing a direct correlation between traffic capacity and the cumulative number of vehicles. Critical flow stage ( ≈ The cumulative number of vehicles on the road network has reached a critical value. Traffic capacity reaches its peak; at this stage, traffic flow tends to saturate, and although the number of vehicles continues to increase, the number of vehicles passing through per unit time tends to stabilize. (Congestion stage) < < At this point, the cumulative number of vehicles on the road network has exceeded the critical value. Excessive traffic volume leads to severe queuing, decreased vehicle speed, and traffic congestion, reducing capacity. (Complete congestion) ≈ At this point, the number of vehicles has reached The road network was almost completely congested, vehicles could not continue to pass, and the traffic capacity was close to zero.

[0082] To obtain different MFD combinations, different traffic demands need to be set for traffic simulation. In order to simulate real traffic flow, a process is set in which the traffic flow slowly increases from a small amount of traffic to a certain level, stabilizes for a period of time, and then decreases. This realistically simulates the dynamic characteristics of traffic flow, which helps to improve the accuracy of the simulation, avoids the impact of sudden changes on the system, and provides a more reliable experimental basis for traffic optimization.

[0083] In S2, a mathematical optimization model is established with the objective of minimizing the number of vehicles in the road network. Specifically:

[0084] In this invention, the selected urban road network area is a relatively congested area. To alleviate the traffic congestion in the area, a transmission flow between the two area boundaries is introduced to minimize the cumulative number of vehicles in both areas. Therefore, the formula for this problem is as follows:

[0085] Constraints:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] in For the final simulation time, for The total number of vehicles in area 1 is the starting and ending point of each time period. for The cumulative number of vehicles with the starting point in Zone 1 and the ending point in Zone 2. for The cumulative number of vehicles with the starting point in Region 2 and the ending point in Region 2. for The starting and ending points of the time are both the cumulative number of vehicles in area 2. , , , It refers to the traffic demand generated within and between different regions. and This represents the proportion of flow control at the boundary. This represents the traffic flow from region i to destination region j. The total capacity within the region is based on... The allocation should be based on the ratio of vehicle density to total vehicle density. The decision is made by formulas (3)-(6). Formulas (3)-(6) are the traffic flow conservation equations, which represent that the change in the number of vehicles in the four states is equal to the traffic demand plus the total inflow minus the total outflow. Formulas (7)-(10) indicate that the total number of vehicles in region 1 and region 2 is greater than or equal to 0 and less than or equal to the cumulative number of vehicles in congestion. Formulas (11)-(12) indicate the boundary control values. , The control range, (13) and (14) represent the control range at the initial time. At that time, the number of vehicles in each of Zone 1 and Zone 2.

[0099] In S3, the Q-Learning algorithm is used, through - The greedy selection strategy chooses the MFD combination with the highest Q value, and optimizes the solution through the MPC controller to obtain the boundary traffic light timings, that is, the traffic light timings of the intersection lanes where vehicles enter area 2 from area 1 or enter area 1 from area 2:

[0100] The goal of Q-Learning is to learn a Q-function, or state-action-value function, to evaluate the value (the cumulative reward that may be obtained in the future) of performing an action in a given state. In this invention, states are not considered; training is conducted solely on historical data through actions. The specific process is as follows:

[0101] 1) First, initialize the Q-table. Combine the MFDs of the two regions as the action set of the Q-table, initialize the Q-value of each action to 0, and set epsilon to the initial exploration probability, with a maximum of 0.9 and a minimum of 0.001. The number of visits for each action is also specified. .

[0102] 2) Update the exploration probability. Epsilon employs linear decay, where the exploration probability gradually decreases as the time step increases, balancing exploration and exploitation in the ε-greedy strategy. This represents the maximum value of epsilon. This is the minimum value of epsilon. This represents the current simulation step count. This is the total simulation time. This is a scaling factor that gradually decreases as the simulation time changes, ensuring that epsilon gradually decreases from its maximum value. When = 0, At this point, epsilon is at its maximum value. hour, At this point, epsilon is at its minimum value. As the time step increases, the algorithm gradually uses the training results to select actions.

[0103]

[0104] 3) Greedy selection strategy. A random probability is generated and compared with the current epsilon: if the probability is less than the epsilon, a random action is explored. Otherwise, the action with the largest Q-value in the current Q-table is selected.

[0105]

[0106] 4) Reward Calculation. The reward is calculated based on the error between the predicted and observed values: the smaller the error, the larger the reward (better control effect). Normalized error is used to avoid scale effects, and a weighted summation is performed. As shown in formula (17), a is the selected action, where This is the observed number of vehicles in area 1. These are the observed values ​​for the number of vehicles in area 2. This is a predicted value for the number of vehicles in area 1. This is the predicted number of vehicles in region 2. The sum of the error values ​​for regions 1 and 2 is calculated, and the error weight is set to 0.5. Multiplying by -1 represents the reward. The smaller the value, the larger the error. When The closer the value is to 0 or is positive, the better the model performance.

[0107]

[0108] 5) Update the Q value. Update the Q value according to the following formula, where... This represents the Q value before the update, i.e., when using the reward value. Before the update, the estimated value stored in the Q table represents the algorithm's assessment of the action. The existing estimates. This represents the updated Q value, i.e., based on the reward value. and number of visits The adjusted new estimate will be written back into the Q table for future action selection and learning.

[0109]

[0110] The simulation time is divided proportionally by simulating traffic flow. At each time step, MFD1 and MFD2, which have the highest Q values, are selected, and rolling optimization is performed using model predictive control. Figure 4As shown, a combination of Q-Learning and MPC controller is used for optimization control in traffic simulation. First, the traffic state between sub-areas is obtained by simulating real traffic flow through SUMO traffic simulation. The Q-Learning part first initializes the parameters, and then updates the exploration probability based on the traffic state obtained from the traffic simulation, the predicted value obtained by the MPC controller, and the Q value. The ε-greedy selection strategy is selected, the reward is calculated, and the Q value is updated. The MPC controller part establishes a prediction model based on the MFD1 and MFD2 functions provided by Q-Learning and the traffic state at the current moment, and predicts the future traffic flow state according to formulas (3)-(6). With minimizing the cumulative number of vehicles between sub-areas as the optimization objective, the optimal control input is calculated using the SLSQP optimization algorithm. and The data is then passed to the traffic simulation module. As the traffic simulation progresses to the next time step, the new state, Q value, and reward are fed back to the Q-Learning and MPC controllers. The system learns and continuously optimizes control as the simulation time changes. Through the closed-loop process of traffic simulation-control-feedback-learning, the system performance is continuously optimized to achieve the goal of alleviating traffic congestion.

[0111] The vehicle flow rate for boundary transfer is calculated based on the boundary control values ​​and the area flow rate calculated by MFD, as shown in the formula:

[0112]

[0113] in yes Time zone Transfer to adjacent sub-region Traffic flow is determined by the density ratio within the area. and boundary control values Decision. To calculate the green light timings at each intersection along the boundary, it is necessary to calculate the traffic flow for each lane, as shown below:

[0114]

[0115] Transfer traffic to the boundary The lanes allocated to the boundary intersection are determined according to lane ratio. Obtain the boundary intersection lane The transfer traffic. The green light time is calculated according to the following formula:

[0116]

[0117] in For saturated flow rate, For the signal period time, for Time Lane The green light timing, among which The following constraints must be met:

[0118]

[0119] If the total allocated green light time exceeds the signal cycle, the green light time for each lane needs to be reduced proportionally.

[0120] In S4, based on real-time traffic flow data, the Cyclic MaxPressure algorithm is used to optimize traffic flow distribution at intersections within congested areas, resulting in internal traffic light timings, i.e., the traffic light timings when vehicles are traveling in their respective lanes within their designated areas. Details are as follows:

[0121] The Cyclic MaxPressure algorithm is a distributed traffic signal control method based on the principle of maximizing pressure, featuring low computational complexity and good adaptive performance. It is an extension and improvement of the classic MaxPressure algorithm; both share the core idea of ​​dynamically optimizing traffic signals by maximizing pressure, but their implementation methods differ. Its basic principle involves calculating the pressure of different signal phases at intersections within a given area, dynamically selecting the signal phase that maximizes the total pressure, and allocating green light time accordingly, thereby achieving local optimization.

[0122] (1) Initialization phase

[0123] A fixed signal period is defined, and a set of possible signal phases is set for each intersection.

[0124] (2) Calculate the pressure

[0125] The key to the algorithm lies in calculating the "Pressure" at each intersection and dynamically adjusting the traffic light status to minimize congestion pressure. The core formula for MaxPressure is the pressure calculation formula, namely:

[0126]

[0127] Among the pressure Reflects the first The degree of traffic flow imbalance under the current signal conditions of each phase; high pressure indicates that there may be vehicle congestion in some lanes of the intersection, requiring more green light time to alleviate traffic congestion. Inflow flow Indicates in Time Lane Traffic flow entering the intersection; a higher inflow rate indicates greater pressure on the intersection from that lane. Outflow rate. Indicates in Other times A higher outflow rate indicates that signal control has effectively reduced congestion in that direction. (Transfer ratio) The drag coefficient between lanes reflects the proportion of traffic flow from one entrance to one exit of an intersection and is an important weight in calculating pressure.

[0128] (3) Select the maximum pressure

[0129] Within each cycle, the signal phase with the highest pressure is selected as the green light signal for the current cycle. The maximum signal phase selected.

[0130]

[0131] (4) Allocating green light time

[0132] Cyclic MaxPressure (CMP) is an extension of the traditional MaxPressure algorithm. It aims to optimize traffic light phase allocation in stages, resulting in more balanced traffic pressure at intersections and avoiding long-term bias towards certain phases. In congested areas, using the Cyclic MaxPressure algorithm to optimize traffic light timing primarily involves dynamically allocating green light time to minimize traffic pressure at each intersection, thereby improving overall traffic flow efficiency in the area. Combined with the MaxPressure algorithm... The proportion of green time for each phase is determined by a probability allocation method. The maximum phase priority is given to allocating green light time:

[0133]

[0134] in For phase The proportion of green light time, For phase Pressure weight, This is an adjustment coefficient used to control the sensitivity of the weight's influence. When When the value is large, the proportion of phases allocated to higher weights is higher; when When the size is smaller, the phase distribution ratio is more uniform. This represents the normalized denominator, ensuring that the sum of all phase proportions is 1. (This is achieved through...) Weights Exponential amplification is applied to enhance the impact of high-pressure-weighted phases on time allocation. Allocation is then performed in conjunction with the green light time of traffic lights, with a signal cycle of [missing information]. Phase The green light timing is:

[0135]

[0136] (5) Periodic updates

[0137] At each new time step, the pressure is recalculated based on the new traffic flow data, and the maximum pressure phase is reselected. As traffic flow changes, the pressure value is adjusted accordingly, so the selection of signal phases and the allocation of green light time also change dynamically.

[0138] The Cyclic MaxPressure algorithm can dynamically respond to traffic flow demands in different directions, effectively alleviating congestion, especially under conditions of uneven flow, by prioritizing time allocation to high-pressure phases. Throughout the simulation, it achieves dynamic optimization of traffic signals, ensuring a reasonable distribution of traffic flow in all directions and avoiding prolonged bias towards certain phases.

[0139] Example 3

[0140] A traffic signal control system combining reinforcement learning and model predictive control includes:

[0141] The MFD fitting module is used to divide the road network area into external regions. and internal area Collect traffic data from various regions and combine it with the traffic demand of each region of the road network to fit different MFD combinations;

[0142] The optimization model building module is used to build a mathematical optimization model based on different MFD combinations, with the goal of minimizing the number of vehicles in the road network area;

[0143] The boundary traffic light timing module is used to combine the Q-Learning algorithm with an established mathematical optimization model. - The greedy selection strategy selects the MFD combination with the highest Q value, and the boundary traffic light timing is obtained by rolling optimization through the MPC controller;

[0144] The internal traffic signal timing module is used to optimize the traffic flow distribution at intersections in congested areas within each region based on collected traffic data and the Cyclic MaxPressure algorithm, thereby obtaining the internal traffic signal timing.

[0145] The innovation of this invention lies in decomposing the optimization problem of regional traffic lights into two main parts: boundary traffic light control and internal traffic light optimization. It specifically employs Q-Learning and Cyclic MaxPressure algorithms, combined with Model Predictive Control (MPC) for optimization. This multi-algorithm collaborative approach not only alleviates congestion in the traffic network to the greatest extent but also improves overall traffic efficiency. Furthermore, this method possesses strong real-time performance and adaptability, enabling rapid adjustment of control strategies based on dynamic traffic flow data, while simultaneously achieving precise control of the overall traffic state through a mathematical optimization model.

[0146] Compared to traditional traffic signal control methods, this invention is significant both theoretically and practically. On one hand, the combination of Q-Learning and Cyclic MaxPressure provides a new approach to intelligent traffic control; on the other hand, regional zoning optimization and multi-algorithm collaborative control make the entire traffic network operate more efficiently and stably.

[0147] In summary, this invention proposes an efficient, flexible, and globally optimized intelligent traffic signal control method. By organically combining reinforcement learning, model predictive control, and zone optimization strategies, it successfully achieves a systematic solution to traffic congestion. This method can not only optimize traffic light timing in real time but also achieve continuous improvement in dynamic traffic environments, demonstrating significant value in alleviating urban traffic pressure and improving road capacity.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A traffic signal control method combining reinforcement learning and model predictive control, characterized by, The method comprises the following steps: Divide the road network area into external area and internal area , collect traffic data of each area, and combine traffic demand of each area of the road network to fit different MFD combinations; a mathematical optimization model is established based on different MFD combinations, aiming to minimize the number of vehicles in the road network region; Based on the established mathematical optimization model, the Q-Learning algorithm is used in combination with - The greedy selection strategy selects the MFD combination with the highest Q value, and the boundary traffic signal timing is obtained by rolling optimization of the MPC controller. wherein the specific method for obtaining the boundary traffic signal timing through rolling optimization of the MPC controller is as follows: select the MFD combination with the highest Q value, and construct a prediction model in combination with the real-time traffic state; predict the traffic flow in the future two regions based on the constructed prediction model; Based on the constructed prediction model, the boundary control input is calculated by the SLSQP optimization algorithm to obtain the boundary control value, with the objective of minimizing the cumulative number of vehicles in the inner region and the outer region ​ based on the boundary control value and the traffic flow in the two regions, calculate the traffic flow transferred at the boundary, and the specific formula is as follows: wherein is time zone traffic flow transferred to adjacent sub-area density ratio in the area boundary control value​ allocate the traffic flow transferred at the boundary to each lane of the boundary intersection, calculate the green light time, and obtain the boundary traffic signal timing; the specific method is as follows: calculate the flow of each lane, and the formula is as follows: The boundary transfer flow is allocated to each lane of the boundary intersection according to the lane proportion The boundary transfer flow is allocated to each lane of the boundary intersection according to the lane proportion The boundary transfer flow is allocated to each lane of the boundary intersection according to the lane proportion The green light time calculation formula of the boundary intersection lane is as follows: wherein is the saturated flow rate, is the signal cycle time, is the green timing for the lane at time instance; based on the collected traffic data, use the Cyclic MaxPressure algorithm to optimize the traffic flow distribution of the intersection in the congestion area of each region, and obtain the internal traffic signal timing; The boundary traffic signal timing is for an outer zone and an inner zone that intersect a lane. The internal traffic signal timing is for the external zone with the internal zone the signal timing of the lanes within each zone.

2. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that, The road network area is divided into an external area and an internal area In the step of collecting traffic data of each area and fitting different MFD combinations combined with traffic demands of each area of the road network, the formula of the MFD is defined as follows: wherein, denotes the region in the trip completion flow, the high order term is used to describe the nonlinear effect of the congestion phase, the quadratic term is used to capture the nonlinear variation of the capacity with density, the linear term reflects the characteristics of vehicles when they are free flowing, , , is a pending parameter, which is obtained by polynomial fitting according to the historical traffic data of traffic flow simulation.

3. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that, in the step of establishing the mathematical optimization model based on different MFD combinations, aiming to minimize the number of vehicles in the road network region, the mathematical optimization model established is as follows: in, For the final simulation time, for Both the starting and ending points of the time are in the external region. The cumulative number of vehicles, for The starting point of time is the outer region The endpoint is the internal area. The cumulative number of vehicles, for The starting point is region 2 and the ending point is the inner region. The cumulative number of vehicles, for Both the starting and ending points of the time are within the internal area. The cumulative number of vehicles.

4. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that, The established mathematical optimization model uses a Q-Learning algorithm combined with In the step of using a Q-Learning algorithm combined with The specific method of updating the Q value by using a greedy selection strategy is as follows: take the MFD combinations of the two regions as the action set of the Q table, and initialize the Q value of each action in the action set to 0; based on the initialized action set, use linear decay for epsilon, update the exploration probability through the epsilon-greedy strategy, obtain the epsilon value at each time step, and update the Q table; generate a probability at random, compare the probability with the epsilon value, if the probability is less than the epsilon value, randomly explore an action; otherwise, select the action with the maximum Q value in the updated Q table; execute the action to obtain the predicted value of the number of vehicles in the two regions, and calculate the reward based on the obtained predicted value and the actual observed value; based on the reward calculation result and the actual observed value of the number of vehicles in the two regions, update the Q value in the Q table using the updating rule of Q-learning.

5. The traffic signal control method combining reinforcement learning and model predictive control according to claim 4, characterized in that, the MFD combination with the highest updated Q value is selected based on the updated Q table.

6. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that, in the step of obtaining the internal traffic signal timing by using the Cyclic MaxPressure algorithm to optimize the traffic flow distribution of the intersection in the congestion area of each region based on the collected traffic data, the specific method used is as follows: determine a fixed signal cycle and a signal phase set of each intersection in the two regions; based on the signal phase set of each intersection, calculate the traffic flow imbalance degree of each intersection through the MaxPressure algorithm as the pressure of each intersection; compare the pressures of each intersection in the same cycle, select the signal phase of the intersection with the maximum pressure, and mark it as the maximum phase signal; dynamically allocate the green light time using the Cyclic MaxPressure algorithm, preferentially allocate the green light time of the maximum phase signal, and obtain the traffic signal timing of the intersection; repeat the above steps according to the real-time traffic flow to dynamically adjust the green light time.

7. The traffic signal control method combining reinforcement learning and model predictive control according to claim 6, characterized in that, The method for dynamically allocating the green light time by using the Cyclic MaxPressure algorithm, preferentially allocating the green light time of the maximum phase signal, and obtaining the traffic signal timing of the intersection is as follows: The green light time proportion of each phase is determined by means of probability distribution The calculation formula is as follows: wherein is the green light time proportion for phase is the pressure weight for phase is the adjustment coefficient for controlling the sensitivity of the weight influence, denotes the normalization denominator ensuring that the sum of all phase proportions is 1;​​ Phase The green light timing is: wherein is the signal period.

8. A traffic signal control system combining reinforcement learning and model predictive control, based on the traffic signal control method combining reinforcement learning and model predictive control according to any one of claims 1-7, characterized in that, Comprise: The MFD fitting module is used to divide the road network area into external regions. and internal area Collect traffic data from various regions and combine it with the traffic demand of each region of the road network to fit different MFD combinations; An optimization model establishing module, configured to establish a mathematical optimization model based on different MFD combinations and aiming to minimize the number of vehicles in a road network region; The boundary traffic signal timing module is used for using a Q-Learning algorithm in combination with - a greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic signal timing through rolling optimization of the MPC controller. An internal traffic signal timing module, configured to use the Cyclic MaxPressure algorithm to optimize the traffic flow distribution of intersections in a congested region based on collected traffic data, and obtain the internal traffic signal timing.

Images