Traffic signal control method and system combining reinforcement learning and model prediction control

Abstract

The invention provides an intelligent adaptive traffic signal control method combining reinforcement learning and model prediction control, and belongs to the technical field of intelligent traffic. In the aspect of boundary signal lamp control, a Q-Learning algorithm is utilized, and an MFD combination selection strategy is continuously optimized through online learning, so that the system can dynamically adapt to different traffic flow modes and change conditions. The Q-Learning algorithm is combined with an epsilon-greedy selection strategy, the control effect is gradually improved through a reward mechanism, and the timing efficiency of the boundary signal lamp is effectively improved. In the aspect of signal optimization of an internal area, a Cyclic MaxPresure algorithm is adopted, signal lamp timing is dynamically adjusted according to real-time traffic flow distribution, the randomness and complexity of traffic flow can be quickly responded, and therefore the congestion condition of a local intersection is effectively relieved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a traffic signal control method and system combining reinforcement learning and model predictive control. Background Art

[0002] With the rapid advancement of urbanization, traffic congestion has become a global problem and an effective solution is urgently needed. Traditional fixed-cycle traffic signal control methods often fail to meet the needs of complex traffic conditions due to their lack of real-time adaptability to dynamic traffic conditions. In contrast, adaptive traffic signal control methods have gradually become an important research field for alleviating urban traffic congestion with their flexibility and dynamic response capabilities. Model Predictive Control (MPC) is a control method based on optimization prediction. By building a systematic prediction model, MPC uses current traffic status and historical data to predict future traffic flow changes and calculates the optimal control input based on the optimization goal. Since traffic signal timing can be dynamically adjusted within a limited future time, MPC has been widely used in traffic signal control. However, MPC is highly dependent on the accuracy of the prediction model in practical applications, but due to the nonlinear, random and time-varying characteristics of traffic flow, relying solely on a fixed model may not achieve the desired control effect. Reinforcement Learning (RL) is an adaptive optimization method based on experimentation and feedback. The agent continuously updates its strategy to maximize long-term rewards by interacting with the environment. Reinforcement learning does not need to rely on an accurate traffic flow model and shows great potential in complex dynamic traffic environments. However, traditional reinforcement learning may face problems such as slow convergence and unstable control effects during the exploration and learning process. Especially in the continuous time and state space of traffic signals, it may be difficult to achieve efficient control by relying solely on reinforcement learning. Therefore, by combining reinforcement learning with model predictive control, dynamic adjustment of urban road network traffic signals can be achieved.

[0003] The current problems with the MPC strategy are: 1) MPC needs to solve an optimization problem in each control cycle, and the optimization problem becomes more complex as the control variables, prediction time domain length, and constraints increase. This computational requirement limits the application of MPC in systems with high real-time requirements or large-scale systems. 2) MPC requires an accurate prediction model to predict the future dynamics of the system. If the prediction model is not accurate, the control effect may be significantly reduced. 3) When the system has significant nonlinearity or high-dimensional state space, the performance of MPC will be limited because traditional optimization algorithms may not be able to efficiently solve nonlinear or high-dimensional problems.

[0004] Problems with reinforcement learning (RL): 1) Reinforcement learning usually requires a large amount of interaction data during training, especially in continuous state and action space problems such as traffic signal control, the sampling efficiency is low, resulting in long training time. 2) RL needs to find a balance between exploring unknown environments and using existing strategies. If exploration is too little, the algorithm may not be able to find the global optimal solution, while too much exploration may lead to unnecessary training overhead. Summary of the invention The purpose of the present invention is to overcome the problems of low efficiency in road network traffic flow distribution and low model prediction accuracy in existing traffic signal control methods, and to provide a traffic signal control method and system combining reinforcement learning and model predictive control.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: A traffic signal control method combining reinforcement learning and model predictive control comprises the following steps: Divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; Based on different MFD combinations, a mathematical optimization model is established with the goal of minimizing the number of vehicles in the road network area; Based on the established mathematical optimization model, the Q-Learning algorithm is combined -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area, and the internal traffic signal timing is obtained; The boundary traffic light timing is the external area With internal area The signal timing of the crossing lanes; The internal traffic light timing is the external area With internal area The traffic lights in the lanes in their respective areas are timed.

[0006] The road network area is divided into external areas and internal areas , collecting traffic data from various areas, and fitting different MFD combinations based on the traffic demand of various areas of the road network, the formula of MFD is defined as follows:

[0007] in, Indicates area middle The trip completion flow, high-order terms Used to describe the nonlinear effects of congestion phases, the quadratic term To capture the nonlinear change of capacity with increasing density, the linear term It reflects the characteristics of vehicles in free flow. , , It is a parameter to be determined, which is obtained by polynomial fitting based on historical traffic data of traffic flow simulation.

[0008] In the step of establishing a mathematical optimization model based on different MFD combinations with the goal of minimizing the number of vehicles in the road network area, the established mathematical optimization model is as follows:

[0009] in, is the final simulation time, for The start and end points of the time are both external areas The cumulative number of vehicles for The time starting point is the external area End point is inner area The cumulative number of vehicles for The starting point of the time is area 2 and the end point is the inner area The cumulative number of vehicles for The start and end points are both inside the area The cumulative number of vehicles.

[0010] The mathematical optimization model based on the establishment is combined with the Q-Learning algorithm - Greedy selection strategy updates the Q value, selects the MFD combination with the highest Q value after the update, and uses the MPC controller to solve the rolling optimization to obtain the boundary traffic light timing step. -The specific method of updating the Q value by the greedy selection strategy is as follows: The MFD combination of the two regions is used as the action set of the Q table, and the Q value of each action in the action set is initialized to 0; Based on the initialized action set, Epsilon uses linear decay to update the exploration probability through the ε-greedy strategy, obtains the epsilon value at each time step and updates the Q table; Randomly generate a probability and compare it with the epsilon value. If the probability is less than the epsilon value, randomly explore an action; otherwise, select the action with the largest Q value in the updated Q table; Execute the above actions to obtain the predicted values ​​of the number of vehicles in the two areas, and calculate the reward between the obtained predicted values ​​and the actual observed values; Based on the reward calculation results and the actual observed values ​​of the number of vehicles in the two areas, the Q-learning update rule is used to update the Q value in the Q table.

[0011] The mathematical optimization model based on the establishment is combined with the Q-Learning algorithm - Greedy selection strategy updates the Q value, selects the MFD combination with the highest Q value after the update, and uses the MPC controller to optimize and solve the problem. In the step of obtaining the timing of the boundary traffic lights, the specific method of using the MPC controller to optimize and solve the problem is as follows: Based on the updated Q table, the MFD combination with the highest Q value is selected, and the prediction model is constructed in combination with the real-time traffic status; Based on the constructed prediction model, the future traffic flow in the two areas is predicted; Based on the constructed prediction model, the internal area and external areas Taking minimizing the cumulative number of vehicles as the goal, the boundary control input is calculated through the SLSQP optimization algorithm to obtain the boundary control value; Based on the boundary control value and the traffic flow in the two areas, the vehicle flow of the boundary transfer is calculated; The traffic flow transferred from the boundary is distributed to each lane of the boundary intersection, the green light time is calculated, and the boundary traffic signal timing is obtained.

[0012] The formula for calculating the traffic flow of the boundary transfer based on the boundary control value and the traffic flow in the two areas is as follows:

[0013] in, yes Time zone Transfer to adjacent subarea traffic flow, is the density ratio in the region, is the boundary control value.

[0014] The method of allocating the traffic flow transferred from the boundary to each lane of the boundary intersection, calculating the green light time, and obtaining the timing of the boundary traffic signal light is as follows: Calculate the flow rate of each lane with the following formula:

[0015] Shifting traffic across borders The lanes assigned to the boundary intersection are divided into lane proportions Get boundary intersection lane The green light time calculation formula is as follows:

[0016] in is the saturation flow rate, is the signal cycle time, for Time Lane Green light timing.

[0017] In the step of optimizing the traffic flow distribution at the intersections of the congested areas in each area based on the collected traffic data using the Cyclic MaxPressure algorithm and obtaining the internal traffic signal light timing, the specific method used is as follows: Determine a fixed signal cycle and a set of signal phases for each intersection in the two areas; Based on the signal phase set of each intersection, the traffic flow imbalance degree of each intersection is calculated by the MaxPressure algorithm as the pressure of each intersection; Compare the pressure of each intersection in the same cycle, select the intersection signal phase with the largest pressure, and record it as the maximum phase signal; Using the Cyclic MaxPressure algorithm to dynamically allocate green light time, giving priority to allocating the green light time of the maximum phase signal, and obtaining the traffic light timing of the intersection; Repeat the above steps according to the real-time traffic flow and dynamically adjust the green light time.

[0018] The method of using the Cyclic MaxPressure algorithm to dynamically allocate green light time, giving priority to allocating the green light time of the maximum phase signal, and obtaining the traffic light timing of the intersection is as follows: The green light time ratio of each phase is determined by probability distribution , the calculation formula is as follows:

[0019] in Phase The green light time ratio, Phase The pressure weight, is the adjustment coefficient, which is used to control the sensitivity of the weight influence. represents the normalized denominator, ensuring that the sum of all phase ratios is 1; Phase The green light timing is:

[0020] in, is the signal period.

[0021] A traffic signal control system combining reinforcement learning and model predictive control, comprising: MFD fitting module, used to divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; The optimization model building module is used to build a mathematical optimization model based on different MFD combinations with the goal of minimizing the number of vehicles in the road network area; The boundary traffic light timing module is used to combine the established mathematical optimization model with the Q-Learning algorithm. -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; The internal traffic light timing module is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area based on the collected traffic data using the Cyclic MaxPressure algorithm to obtain the internal traffic light timing.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a traffic signal control method and system combining reinforcement learning and model predictive control, comprising the following steps: dividing the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of each area of ​​the road network; based on different MFD combinations, establish a mathematical optimization model with the goal of minimizing the number of vehicles in the road network area; based on the established mathematical optimization model, use the Q-Learning algorithm combined with -The greedy selection strategy selects the MFD combination with the highest Q value, and solves it through rolling optimization by the MPC controller to obtain the boundary traffic light timing; based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area to obtain the internal traffic light timing. By dividing the traffic road network into regions, different optimization strategies are adopted for the boundary area and the internal area respectively, and the partition optimization is achieved, thereby significantly improving the traffic flow distribution efficiency of the entire road network. In terms of boundary signal light control, the present invention uses the Q-Learning algorithm to continuously optimize the MFD combination selection strategy through online learning, so that the system can dynamically adapt to different traffic flow patterns and changes. In terms of signal optimization in the internal area, the present invention adopts the Cyclic MaxPressure algorithm to dynamically adjust the signal light timing according to the real-time traffic flow distribution, so that it can quickly respond to the randomness and complexity of the traffic flow, improve the accuracy of the model prediction, and effectively alleviate the congestion at the local intersection.

[0023] Furthermore, the Q-Learning algorithm combines - Greedy selection strategy, which gradually improves the control effect through the reward mechanism, effectively improving the timing efficiency of boundary traffic lights.

[0024] Furthermore, the present invention alleviates local congestion, avoids the spread of congestion to other areas, effectively reduces the total number of vehicles in the traffic network, and improves the overall traffic capacity of the road network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the present invention; Figure 2 A schematic diagram of a two-region road network based on SUMO simulation of the present invention; Figure 3 It is the macro basic diagram of the present invention; Figure 4 The present invention combines Q-Learning and MPC controller for optimal control in traffic simulation. DETAILED DESCRIPTION

[0026] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0027] Example 1 like Figure 1 As shown, a traffic signal control method combining reinforcement learning and model predictive control includes the following steps: S1: Divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; S2: Based on different MFD combinations, a mathematical optimization model is established with the goal of minimizing the number of vehicles in the road network area; S3: Based on the established mathematical optimization model, the Q-Learning algorithm is combined -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; S4: Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area to obtain the internal traffic signal light timing; The boundary traffic light timing is the external area With internal area The signal timing of the crossing lanes; The internal traffic light timing is the external area With internal area The traffic lights in the lanes in their respective areas are timed.

[0028] Example 2 In S1, Figure 2 As shown in the figure, the road network area is divided into zones, and the basic parameters such as the length of each section and the number of lanes of the road network in the study area are set according to the actual situation. In this study, the road network is divided into two homogeneous areas with different characteristics, forming a heterogeneous transportation network. One area is the external area. (area 1), the other is the inner area (Area 2). The road network is divided into two sub-areas with a containment relationship. The traffic flow between the two sub-areas is mainly exchanged through 12 intersections. , where each zone has a good MFD, and for a two-zone system, there are two endogenous traffic demands, In, use (veh / s) means, In, use (veh / s) means, and The generated destination is and Two exogenous traffic demands (i.e., border traffic) are represented by and (veh / s). Therefore, there are four different state variables (veh), where It means that Time zone To destination The total number of vehicles, similarly (veh) means in Time zone The total cumulative number of vehicles in .

[0029] The macro basic diagram MFD uses (veh / s) means that it indicates the area middle The trip completion flow is as follows: Figure 2 As shown in Figure 2, the shape of MFD is approximately an asymmetric single-peak curve tilted to the right, so it can be used to The third-order function is defined as follows:

[0030] Among them, the higher-order terms Used to describe the trend of decreasing capacity when the density is high, that is, the nonlinear effect of the congestion stage, the quadratic term Used to capture the nonlinear change in capacity as density increases, and is suitable for medium density ranges. It reflects the direct impact of the number of vehicles on traffic capacity when the density is low, that is, the characteristics of vehicles flowing freely. , , is a parameter to be determined, obtained through polynomial fitting based on historical traffic data of traffic flow simulation, and used to describe some characteristics of different areas. Figure 3 As shown in the figure, the horizontal axis (accumulated number of vehicles on the road network) represents the total number of vehicles on all roads in the area, that is, the overall quantitative indicator of traffic density, usually in units of vehicles (veh). is the critical cumulative number of vehicles, corresponding to the maximum traffic capacity. The vertical axis is the cumulative number of vehicles in congestion, which indicates the total number of vehicles in complete congestion. It indicates the number of vehicles that can complete the passage in a region per unit time (veh / s), reflecting the traffic capacity of the road network. MFD usually consists of a peak and two descending stages, corresponding to the free flow, critical flow and congested flow states: free flow stage ( <= ): There are relatively few vehicles in the road network, and the interference between vehicles is relatively small. The traffic capacity will increase with the increase in the number of vehicles, and the traffic capacity is proportional to the cumulative number of vehicles. Critical flow stage ( ≈ ): The cumulative number of vehicles on the road network reaches a critical value , the traffic capacity reaches its peak; at this stage, the traffic flow tends to be saturated. Although the number of vehicles continues to increase, the number of vehicles passing through per unit time tends to be stable. < < ): At this time, the cumulative number of vehicles on the road network has exceeded the critical value. There are too many vehicles, resulting in serious queuing in the road network, a decrease in vehicle speed, traffic congestion, and reduced traffic capacity. Complete congestion state ( ≈ ): At this time, the number of vehicles reaches , the road network traffic is basically completely congested, vehicles can no longer pass, and the traffic capacity approaches zero.

[0031] In order to obtain different MFD combinations, different traffic demands need to be set for traffic simulation. In order to simulate real traffic flow, a process is set in which the traffic flow slowly increases from a small amount of traffic to a certain amount of traffic, stabilizes for a period of time, and then decreases. This can truly simulate the dynamic characteristics of traffic flow, which helps to improve the accuracy of the simulation and avoid the impact of sudden changes on the system. At the same time, it provides a more reliable experimental basis for traffic optimization.

[0032] In S2, a mathematical optimization model is established with the goal of minimizing the number of vehicles on the road network. Specifically: In the present invention, the selected urban road network area is a relatively congested area. In order to alleviate the traffic conditions in the congested area, the transmission flow between the boundaries of the two areas is introduced to minimize the cumulative number of vehicles in the two areas. Therefore, the formula for this problem is as follows: Constraints:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] in is the final simulation time, for The cumulative number of vehicles whose starting and ending points are both in area 1 at the time, for The cumulative number of vehicles starting from area 1 and ending at area 2 at the time, for The cumulative number of vehicles whose starting point is area 2 and whose end point is area 2, for The starting point and the end point at each time are the cumulative number of vehicles in area 2. , , , is the traffic demand generated within and between regions, and It represents the flow control ratio of the boundary. represents the flow from area i to destination area j. The total capacity within the area is based on The allocation ratio should be the ratio of vehicle density to total vehicle density. Formulas (3)-(6) are the traffic flow conservation equations, which represent that the change in the number of vehicles in the four states is equal to the traffic demand plus the total inflow minus the total outflow. Formulas (7)-(10) indicate that the total number of vehicles in area 1 and area 2 is greater than or equal to 0 and less than or equal to the cumulative number of vehicles in congestion. (11)-(12) represent the boundary control value , The control range of (13) and (14) indicates that at the initial moment When , the number of vehicles in area 1 and area 2 respectively.

[0045] In S3, the Q-Learning algorithm is used. -The greedy selection strategy selects the MFD combination with a high Q value, and optimizes the solution through the MPC controller to obtain the boundary traffic light timing, that is, the traffic light timing of the cross lane when the vehicle enters area 2 from area 1 or from area 2 to area 1: The goal of Q-Learning is to learn a Q function, i.e., a state-action value function, which is used to evaluate the value of performing an action in a certain state (the cumulative reward that may be obtained in the future). In this invention, the state is not considered, and only the historical data is trained through actions. The specific process is as follows: 1) First, initialize the Q table, and use the MFD combination of the two regions as the action set of the Q table. Initialize the Q value of each action to 0, epsilon is the initial exploration probability, the maximum is 0.9, the minimum is 0.001, and the number of visits for each action is .

[0046] 2) Update the exploration probability. Epsilon uses linear decay. As the time step increases, the exploration probability gradually decreases to achieve a balance between exploration and utilization in the ε-greedy strategy. is the maximum value of epsilon, is the minimum value of epsilon, is the current simulation step number, is the total simulation time. is a scaling factor that decreases gradually with simulation time, ensuring that epsilon decreases gradually from its maximum value. =0, , at this time epsilon is the maximum value, when hour, , at this time, epsilon is the minimum value. As the time step advances, the algorithm gradually uses the training results to select actions.

[0047]

[0048] 3) ϵ-greedy selection strategy. Randomly generate a probability and compare it with the current epsilon: if it is less than epsilon, randomly explore an action. Otherwise, select the action with the largest Q value in the current Q-table.

[0049]

[0050] 4) Reward calculation. The reward is calculated based on the error between the predicted value and the observed value: the smaller the error, the greater the reward (better control effect). Normalized error is used to avoid scale effect and weighted sum is used. As shown in formula (17), a is the selected action, where is the observed value of the number of vehicles in region 1, is the observed value of the number of vehicles in area 2, is the predicted value of the number of vehicles in area 1, is the predicted value of the number of vehicles in area 2. Calculate the sum of the error values ​​of area 1 and area 2, and set the error weight to 0.5. Multiplying by -1 represents the reward The smaller the error, the greater the error. The closer it is to 0 or positive, the better the model performance.

[0051]

[0052] 5) Update the Q value. Update the Q value according to the following formula, where Represents the Q value before the update, that is, when using the reward value Before updating, the estimated value stored in the Q table represents the algorithm's response to the action existing estimates. Represents the updated Q value, that is, according to the reward value and number of visits The adjusted new estimate is written back into the Q-table for future action selection and learning.

[0053]

[0054] The simulation time is divided into equal proportions by simulating traffic flow, and MFD1 and MFD2 with the highest Q value are selected at each time step, and rolling optimization is performed through model predictive control. Figure 4 As shown in the figure, Q-Learning and MPC controller are combined for optimal control in traffic simulation. First, the real traffic flow is simulated by SUMO traffic simulation to obtain the traffic status between sub-areas. The Q-Learning part first initializes the parameters, updates the exploration probability according to the traffic status obtained by traffic simulation, the predicted value obtained by the MPC controller, and the Q value, performs ϵ-greedy selection strategy selection, calculates the reward and updates the Q value. The MPC controller part establishes a prediction model based on the MFD1 and MFD2 functions provided by Q-Learning and the current traffic status, and predicts the future traffic flow status according to formulas (3)-(6). Taking the minimization of the cumulative number of vehicles between sub-areas as the optimization goal, the SLSQP optimization algorithm is used to calculate the optimal control input. and , pass it to the traffic simulation module. As the traffic simulation runs to the next time step, the new state, Q value, and reward are fed back to the Q-Learning and MPC controllers. As the simulation time changes, gradual learning and rolling optimization control are performed. Through the closed-loop process of traffic simulation-control-feedback-learning, the system performance is continuously optimized to achieve the goal of alleviating traffic congestion.

[0055] The vehicle flow of boundary transfer is calculated based on the boundary control value and the regional flow calculated by MFD, as shown in the formula:

[0056] in yes Time zone Transfer to adjacent subarea The traffic flow is determined by the density ratio in the area and boundary control values In order to calculate the green light timing of each intersection on the border, it is necessary to calculate the flow of each lane, as shown below:

[0057] Shifting traffic across borders The lanes assigned to the boundary intersection are divided into lane proportions Get boundary intersection lane The green light time is calculated according to the following formula:

[0058] in is the saturation flow rate, is the signal cycle time, for Time Lane The green light timing of The following constraints need to be met:

[0059] If the total allocated green time exceeds the signal cycle, the green time of each lane needs to be reduced proportionally.

[0060] In S4, based on real-time traffic flow data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at the intersection of the internal congested area, and the internal traffic light timing is obtained, that is, the signal light timing when vehicles are driving in the internal lanes of their respective areas. The details are as follows: The Cyclic MaxPressure algorithm is a distributed traffic signal control method based on the principle of pressure maximization, with low computational complexity and good adaptive performance. The Cyclic MaxPressure algorithm is an extension and improvement of the classic MaxPressure algorithm. The core idea of ​​both is to achieve dynamic optimization of traffic signals by maximizing pressure, but there are differences in the implementation methods. The basic principle is to calculate the pressure of different signal phases at intersections within the area, dynamically select the signal phase that maximizes the total pressure, and allocate the green light time, thereby achieving local optimization.

[0061] (1) Initialization phase A fixed signal cycle is determined and a possible signal phase set is set for each intersection.

[0062] (2) Calculation of pressure The key to the algorithm is to calculate the "pressure" of each intersection and minimize the congestion pressure by dynamically adjusting the status of the traffic lights. The core formula of MaxPressure is the pressure calculation formula, which is:

[0063] The pressure Reflects the The degree of traffic flow imbalance under the current signal state of each phase. High pressure indicates that some lanes of the intersection may have vehicles stuck and more green light time needs to be allocated to ease traffic congestion. Indicated in Time Lane Traffic flow entering the intersection. The higher the inflow flow, the greater the pressure on the intersection from that lane. Indicated in Other time The amount of traffic leaving the intersection through the signal. A higher outflow means that the signal control has effectively reduced the pressure in that direction. It represents the resistance coefficient between lanes. It reflects the proportion of traffic flow from a certain entrance to a certain exit of the intersection and is an important weight for calculating pressure.

[0064] (3) Select the maximum pressure In each cycle, the signal phase with the largest pressure is selected as the green light signal of the current cycle, where is the selected maximum signal phase.

[0065]

[0066] (4) Allocate green light time Cyclic MaxPressure (CMP) is an extension of the traditional MaxPressure algorithm. It aims to optimize the distribution of traffic light phases in stages and cycles, so that the pressure in the intersection is more balanced and the phenomenon of long-term bias towards certain phases is avoided. Cyclic MaxPressure algorithm is used to optimize traffic light timing in congested areas, mainly by dynamically allocating green light time to minimize the traffic pressure at each intersection, thereby improving the overall traffic flow efficiency of the area. , the green light time ratio of each phase is determined in a probabilistic manner , the maximum phase priority allocation green light time:

[0067] in Phase The green light time ratio, Phase The pressure weight, is the adjustment coefficient, which is used to control the sensitivity of the weight influence. When it is larger, the high-weight phase allocation ratio is higher; when When it is smaller, the distribution ratio of each phase is more uniform. represents the normalized denominator, ensuring that the sum of all phase ratios is 1. Weight Exponential amplification is performed to enhance the impact of the phase with high pressure weight on time allocation. The green light time of the traffic light is allocated, and the signal cycle is ,Phase The green light timing is:

[0068] (5) Periodic updates At each new time step, the pressure is recalculated based on the new traffic flow data and the maximum pressure phase is reselected. As traffic flow changes, the pressure value is adjusted accordingly, so the choice of signal phase and green light time allocation will also change dynamically.

[0069] The Cyclic MaxPressure algorithm can dynamically respond to traffic flow demands in different directions and effectively alleviate congestion, especially in the case of uneven traffic flow, by giving priority to high-pressure phases. Dynamic optimization of traffic signals is achieved throughout the simulation to ensure the reasonable distribution of traffic flow in all directions and avoid long-term bias towards certain phases.

[0070] Example 3 A traffic signal control system combining reinforcement learning and model predictive control, comprising: MFD fitting module, used to divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; The optimization model building module is used to build a mathematical optimization model based on different MFD combinations with the goal of minimizing the number of vehicles in the road network area; The boundary traffic light timing module is used to combine the established mathematical optimization model with the Q-Learning algorithm. -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; The internal traffic light timing module is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area based on the collected traffic data using the Cyclic MaxPressure algorithm to obtain the internal traffic light timing.

[0071] The innovation of this invention lies in decomposing the optimization problem of regional traffic lights into two parts: boundary light control and internal light optimization. The Q-Learning and Cyclic MaxPressure algorithms are used in a targeted manner, and the optimization solution is combined with model predictive control (MPC). This multi-algorithm collaborative approach can not only alleviate the congestion problem in the traffic network to the greatest extent, but also improve the efficiency of global traffic operation. In addition, this method has strong real-time and adaptability, and can quickly adjust the control strategy according to dynamic traffic flow data, while achieving precise control of the global traffic status through mathematical optimization models.

[0072] Compared with the traditional traffic signal control method, this invention has important significance in both theory and practical application. On the one hand, the combination of Q-Learning and Cyclic MaxPressure provides a new idea for intelligent traffic control; on the other hand, regional partition optimization and multi-algorithm collaborative control make the operation of the entire traffic network more efficient and stable.

[0073] In summary, the present invention proposes an efficient, flexible and globally optimized intelligent traffic signal control method, which successfully achieves a systematic solution to the traffic congestion problem through the organic combination of reinforcement learning, model predictive control and partition optimization strategy. This method can not only optimize the timing of traffic lights in real time, but also achieve continuous improvement in a dynamic traffic environment, which has significant value in alleviating urban traffic pressure and improving road capacity.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A traffic signal control method combining reinforcement learning and model predictive control, characterized in that: The steps include: Divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; Based on different MFD combinations, a mathematical optimization model is established with the goal of minimizing the number of vehicles in the road network area; Based on the established mathematical optimization model, the Q-Learning algorithm is combined -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; Based on the collected traffic data, the Cyclic MaxPressure algorithm is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area, and the internal traffic signal timing is obtained; The boundary traffic light timing is for the external area With internal area The signal timing of the crossing lanes; The internal traffic light timing is the external area With internal area The traffic lights in the lanes in their respective areas are timed.

2. A traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that: The road network area is divided into external areas and internal areas , collecting traffic data from various areas, and fitting different MFD combinations based on the traffic demand of various areas of the road network, the formula of MFD is defined as follows: in, Indicates area middle The trip completion flow, high-order terms Used to describe the nonlinear effects of congestion phases, the quadratic term To capture the nonlinear change of capacity with increasing density, the linear term It reflects the characteristics of vehicles in free flow. , , It is a parameter to be determined, which is obtained by polynomial fitting based on historical traffic data of traffic flow simulation.

3. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that: In the step of establishing a mathematical optimization model based on different MFD combinations with the goal of minimizing the number of vehicles in the road network area, the established mathematical optimization model is as follows: in, is the final simulation time, for The start and end points of the time are both external areas The cumulative number of vehicles for The time starting point is the external area End point is inner area The cumulative number of vehicles for The starting point of the time is area 2 and the end point is the inner area The cumulative number of vehicles for The start and end points are both inside the area The cumulative number of vehicles.

4. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that: The mathematical optimization model based on the establishment is combined with the Q-Learning algorithm - Greedy selection strategy updates the Q value, selects the MFD combination with the highest Q value after the update, and uses the MPC controller to solve the rolling optimization to obtain the boundary traffic light timing step. -The specific method of updating the Q value by the greedy selection strategy is as follows: The MFD combination of the two regions is used as the action set of the Q table, and the Q value of each action in the action set is initialized to 0; Based on the initialized action set, Epsilon uses linear decay to update the exploration probability through the ε-greedy strategy, obtains the epsilon value at each time step and updates the Q table; Randomly generate a probability and compare it with the epsilon value. If the probability is less than the epsilon value, randomly explore an action; otherwise, select the action with the largest Q value in the updated Q table; Execute the above actions to obtain the predicted values ​​of the number of vehicles in the two areas, and calculate the reward between the obtained predicted values ​​and the actual observed values; Based on the reward calculation results and the actual observed values ​​of the number of vehicles in the two areas, the Q-learning update rule is used to update the Q value in the Q table.

5. The traffic signal control method combining reinforcement learning and model predictive control according to claim 4, characterized in that: The mathematical optimization model based on the establishment is combined with the Q-Learning algorithm - Greedy selection strategy updates the Q value, selects the MFD combination with the highest Q value after the update, and uses the MPC controller to optimize and solve the problem. In the step of obtaining the timing of the boundary traffic lights, the specific method of using the MPC controller to optimize and solve the problem is as follows: Based on the updated Q table, the MFD combination with the highest Q value is selected, and the prediction model is constructed in combination with the real-time traffic status; Based on the constructed prediction model, the future traffic flow in the two areas is predicted; Based on the constructed prediction model, the internal area and external areas Taking minimizing the cumulative number of vehicles as the goal, the boundary control input is calculated through the SLSQP optimization algorithm to obtain the boundary control value; Based on the boundary control value and the traffic flow in the two areas, the vehicle flow of the boundary transfer is calculated; The traffic flow transferred from the boundary is distributed to each lane of the boundary intersection, the green light time is calculated, and the boundary traffic signal timing is obtained.

6. The traffic signal control method combining reinforcement learning and model predictive control according to claim 5, characterized in that: The formula for calculating the traffic flow of the boundary transfer based on the boundary control value and the traffic flow in the two areas is as follows: in, yes Time zone Transfer to adjacent subarea traffic flow, is the density ratio in the region, is the boundary control value.

7. The traffic signal control method combining reinforcement learning and model predictive control according to claim 6, characterized in that: The method of allocating the traffic flow transferred from the boundary to each lane of the boundary intersection, calculating the green light time, and obtaining the timing of the boundary traffic signal light is as follows: Calculate the flow rate of each lane using the following formula: Shifting traffic across borders The lanes assigned to the boundary intersection are divided into lane proportions Get boundary intersection lane The green light time calculation formula is as follows: in is the saturation flow rate, is the signal cycle time, for Time Lane Green light timing.

8. The traffic signal control method combining reinforcement learning and model predictive control according to claim 1, characterized in that: In the step of optimizing the traffic flow distribution at the intersections of the congested areas in each area based on the collected traffic data using the Cyclic MaxPressure algorithm and obtaining the internal traffic signal light timing, the specific method used is as follows: Determine a fixed signal cycle and a set of signal phases for each intersection in the two areas; Based on the signal phase set of each intersection, the traffic flow imbalance degree of each intersection is calculated by the MaxPressure algorithm as the pressure of each intersection; Compare the pressure of each intersection in the same cycle, select the intersection signal phase with the largest pressure, and record it as the maximum phase signal; Using the Cyclic MaxPressure algorithm to dynamically allocate green light time, giving priority to allocating the green light time of the maximum phase signal, and obtaining the traffic light timing of the intersection; Repeat the above steps according to the real-time traffic flow and dynamically adjust the green light time.

9. The traffic signal control method combining reinforcement learning and model predictive control according to claim 8, characterized in that: The method of using the Cyclic MaxPressure algorithm to dynamically allocate green light time, giving priority to allocating the green light time of the maximum phase signal, and obtaining the traffic light timing of the intersection is as follows: The green light time ratio of each phase is determined by probability distribution , the calculation formula is as follows: in Phase The green light time ratio, Phase The pressure weight, is the adjustment coefficient, which is used to control the sensitivity of the weight influence. represents the normalized denominator, ensuring that the sum of all phase ratios is 1; Phase The green light timing is: in, is the signal period.

10. A traffic signal control system combining reinforcement learning and model predictive control, characterized in that: include: MFD fitting module, used to divide the road network area into external areas and internal areas , collect traffic data from various areas, and fit different MFD combinations based on the traffic demand of various areas of the road network; The optimization model building module is used to build a mathematical optimization model based on different MFD combinations with the goal of minimizing the number of vehicles in the road network area; The boundary traffic light timing module is used to combine the established mathematical optimization model with the Q-Learning algorithm. -The greedy selection strategy selects the MFD combination with the highest Q value, and obtains the boundary traffic light timing through the rolling optimization solution of the MPC controller; The internal traffic light timing module is used to optimize the traffic flow distribution at the intersections of the internal congested areas of each area based on the collected traffic data using the Cyclic MaxPressure algorithm to obtain the internal traffic light timing.

Images