Urban traffic optimization and planning system based on reinforcement learning

Abstract

The present invention discloses an urban traffic optimization and planning system based on reinforcement learning, which relates to the field of computer information technology. The system includes: a traffic data acquisition unit, a reinforcement learning processing unit and a traffic planning unit; the traffic data acquisition unit is used to regard the target city traffic as a network, each intersection as a node, and construct a nonlinear traffic flow dynamics model to update the multidimensional tensor at each time step; the reinforcement learning processing unit is used to use the signal timing matrix of the traffic lights of the target city traffic on the road section between each two nodes as the state space, and calculate the corresponding action space when the target reward function is the maximum value; the traffic planning unit is used to complete the planning of the target city traffic according to the action space of each road section. The present invention significantly improves the global optimization ability, multi-objective optimization ability, adaptive learning ability, data processing ability, security and interpretability of traffic signal control.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to an urban traffic optimization and planning system based on reinforcement learning. Background Art

[0002] In the current urban traffic management system, with the continuous acceleration of urbanization and the rapid growth of the number of motor vehicles, the urban traffic network is facing an increasingly serious congestion problem. This problem not only affects the travel efficiency and quality of life of citizens, but also leads to increased energy consumption and increased environmental pollution. Therefore, how to optimize the urban traffic signal control strategy and improve the traffic efficiency of traffic flow has become one of the core challenges of urban traffic management. In this context, the development of intelligent transportation systems (ITS) has become an effective way to meet this challenge. Intelligent transportation systems provide a series of innovative solutions for traffic signal control and traffic flow management by integrating information technology, communication technology, electronic control technology and artificial intelligence technology. However, there are still some problems that need to be solved in the practical application of existing technologies.

[0003] At present, the mainstream methods of urban traffic signal control mainly include timing control, adaptive control and rule-based control. The traditional timing control method pre-sets a fixed signal light timing scheme and adjusts it in different time periods according to the pre-calculated traffic flow data. The advantage of this method is that it is simple and easy to implement and is suitable for scenes with relatively stable traffic flow. However, its limitations are also very obvious: once the traffic flow changes suddenly, the fixed signal light timing scheme cannot respond in time, resulting in a significant reduction in the optimization effect of traffic flow, and may even cause more serious traffic congestion.

[0004] To solve this problem, adaptive control methods have emerged. Adaptive control dynamically adjusts the signal timing scheme by collecting real-time traffic data, such as vehicle queue length, vehicle passing rate and average travel time, so as to better adapt to the current traffic conditions. SCATS (Sydney Coordinated Adaptive Traffic System) and SCOOT (Distributed Coordinated Optimized Traffic Signal Control System) are two typical adaptive traffic signal control systems. They calculate the optimal signal timing scheme based on real-time data through optimization algorithms. However, the optimization process of these systems depends on the statistical characteristics of traffic flow, and usually adopts local optimization strategies and lacks the ability of global optimization. When facing complex multi-intersection traffic networks, this limitation makes it difficult to achieve global optimal traffic signal control when dealing with changing traffic conditions. In addition, rule-based traffic signal control methods adjust the timing and phase switching of signal lights through pre-defined rule sets (such as traffic flow thresholds, vehicle priority rules, etc.). Although this method has certain flexibility and operability, its optimization effect is highly dependent on the design quality of the rule set. In actual traffic environments, the changes in traffic flow are highly uncertain and dynamic, and it is difficult for pre-defined rule sets to fully cover all possible traffic conditions. Therefore, the control strategies of rule-based methods are significantly insufficient in robustness and adaptability when facing emergencies (such as traffic accidents, road construction, etc.) or abnormal traffic flows. Summary of the invention

[0005] The purpose of this invention is to provide an urban traffic optimization and planning system based on reinforcement learning, which significantly improves the global optimization capability, multi-objective optimization capability, adaptive learning capability, data processing capability, security and interpretability of traffic signal control. It not only improves the overall traffic efficiency of the urban traffic network, reduces congestion and environmental pollution, but also enhances the stability and security of the system, and has a wide range of practical application prospects.

[0006] In order to solve the above technical problems, the present invention provides an urban traffic optimization and planning system based on reinforcement learning, the system comprising: a traffic data acquisition unit, a reinforcement learning processing unit and a traffic planning unit; the traffic data acquisition unit is used to regard the target city traffic as a network, each intersection as a node, obtain the traffic data of the target city traffic on the road section between each two nodes, assign a weight to each road section, define the traffic network state of each road section as a multidimensional tensor, and construct a nonlinear traffic flow dynamics model to update the multidimensional tensor at each time step; the reinforcement learning processing unit is used to use the signal timing matrix of the traffic lights on the road section between each two nodes of the target city traffic as the state space, define the target reward function of the reinforcement learning model, and calculate the corresponding action space when the target reward function is the maximum value through the pre-trained reinforcement learning model under the state space corresponding to each time step; the traffic planning unit is used to control the operation of the traffic lights on each road section according to the action space of each road section to complete the planning of the target city traffic.

[0007] Furthermore, the traffic data of the road section (i, j) at time t at least includes: vehicle density Average vehicle speed Traffic flow and average travel time The multidimensional tensor is:

[0008]

[0009] Where N is the number of nodes; i and j are both subscript indices with positive integer values; Represents the field of real numbers.

[0010] Furthermore, the initial density of each road segment (i, j) is set to and the initial velocity is Define the time step Δt, and for each time step t=0, Δt, 2Δt, ..., divide each road segment (i, j) into M ij equal-length sub-segments; for each sub-segment m=1, 2, ..., M ij , the vehicle density, average vehicle speed, traffic flow and average travel time are updated using a preset nonlinear traffic flow dynamics model; the nonlinear traffic flow dynamics model is expressed using the following formula:

[0011]

[0012] in, is the maximum vehicle density of road section (i, j); is the maximum traffic flow of road section (i, j); is the free flow speed of the road section (i, j), which represents the average speed of vehicles when the road is unobstructed; α ij , β ij and γ ij are all characteristic index parameters of the preset road section (i, j); D ij is the diffusion coefficient of the road section (i, j); is the entrance flow of section (i, j) at time t; is the exit flow of section (i, j) at time t.

[0013] Furthermore, the vehicle density, average vehicle speed, traffic flow and average travel time are updated through the following formula:

[0014]

[0015]

[0016] in, For the time t k+1 vehicle density; For the time t k+1 The average speed of vehicles; For the time t k+1 traffic flow; For the time t k+1 Average travel time; For the time t k vehicle density; For the time t k The average speed of vehicles; L ij is the length of the road segment (i, j).

[0017] Furthermore, the state space is a multidimensional tensor:

[0018]

[0019] Where K is the maximum number of phases at each intersection; is the green light time vector of node i at time t; is the phase order vector of node i at time t; is the offset and switching time vector of node i at time t;

[0020]

[0021] in is the green time of the sth phase of node i at time t;

[0022]

[0023] in represents the phase number of the sth execution sequence of node i at time t;

[0024]

[0025] in is the offset of node i relative to the system clock at time t; is the yellow light time of node i in the kth phase at time t; is the all-red time of node i at time t.

[0026] Furthermore, the target reward function R(S t , A t ) is expressed using the following formula:

[0027] R(S t , A t )=w flow ·R flow (S t , A t )+w stability ·R stability (S t , A t );

[0028] Among them, R flow (S t , A t ) is the traffic efficiency objective reward function; R stability (S t , A t ) is the traffic stability objective reward function; w flow is the preset traffic efficiency weight; w stability is the preset traffic stability weight; where w flow +w stability =1.

[0029] Furthermore, the traffic efficiency objective reward function is expressed using the following formula:

[0030]

[0031] in, is the sum of green light time of road section (i, j) at time t; is the optimal vehicle density of the preset road section (i, j); σ ij is the density tolerance parameter of the preset road section (i, j); is the most critical speed of the road section (i, j), which means the average speed corresponding to the road section (i, j) when the maximum traffic flow of the road section (i, j) is lower than the preset traffic critical value; is the maximum allowed speed of road section (i, j); The signal offset of the road section (i, j) at time t; |·| is the operation of finding the modulus of the vector.

[0032] Furthermore, the traffic stability objective reward function is expressed using the following formula:

[0033]

[0034] Among them, w ij is the weight of the preset road section (i, j); is the average delay of link (i, j) at time t; is the entropy of all output flows of node i; H max is the maximum possible entropy of all output flows of node i when the distribution is completely uniform; is the queue length of section (i, j) at time t; is the target total queue length of node i; σ l is the preset queue length tolerance parameter;

[0035] Furthermore, the parameter θ update formula of the reinforcement learning model is:

[0036]

[0037] Among them, η is the learning rate, c is the gradient clipping threshold, λ is the weight decay coefficient, λ1 is the L2 regularization coefficient; θ new is the updated reinforcement learning parameter; L(θ) is the loss function of reinforcement learning; clip(·, -c, c) is the gradient clipping function, which limits the gradient value to the range of [-c, c], and c is the preset value; is the gradient operator; Represents a replay buffer of experiences learned from reinforcement learning Sampling conversion (S t , A t , R(S t , A t ), A t ′) take the average, A t ' is the next action; represents the square of the L2 norm; θ - is the target parameter of reinforcement learning; Q(S t , A t ′;θ - ) is a given state S t and action A t ′, using parameter θ - The predicted Q value.

[0038] The urban traffic optimization and planning system based on reinforcement learning of the present invention has the following beneficial effects:

[0039] First, the present invention significantly improves the global optimization capability of traffic signal control. Traditional traffic signal control systems usually rely on preset rules or strategies based on local optimization, such as timing control or adaptive control. These methods often lack a global perspective when facing complex and changeable urban traffic networks, and it is difficult to effectively coordinate traffic flows between multiple intersections. However, through the introduction of the reinforcement learning model in the present invention, the system can dynamically optimize the entire traffic network from a global perspective. Through continuous interaction and learning with the environment, the reinforcement learning model can adaptively adjust the timing and phase of traffic lights, taking into account the mutual influence and synergy between each intersection, thereby achieving global optimization of the entire traffic network. Especially in large-scale, multi-intersection traffic networks, the present invention can significantly reduce the average waiting time of vehicles, improve road capacity, and reduce the occurrence of traffic congestion.

[0040] Secondly, the present invention has a powerful multi-objective optimization capability. Most existing traffic signal control methods focus on optimizing a single objective, such as minimizing vehicle delays or maximizing traffic flow, while ignoring other important traffic management objectives. The present invention comprehensively considers multiple optimization objectives such as traffic efficiency and traffic stability by designing a multi-objective reward function. Specifically, the system not only focuses on maximizing traffic flow and minimizing vehicle waiting time, but also considers multiple factors such as the balance of traffic flow, the smoothness of vehicle driving, and the reduction of emissions. Through this multi-objective optimization strategy, the present invention can provide a more flexible and efficient signal light control strategy under different traffic conditions to meet the multiple needs of urban traffic management. This multi-objective optimization method not only improves the efficiency of the overall traffic system, but also effectively reduces environmental pollution, improves traffic safety and driving experience.

[0041] Third, the present invention enhances the adaptive learning ability and robustness of the system by introducing deep reinforcement learning technology. In the actual urban traffic environment, traffic flow and patterns are highly dynamic and uncertain, and traditional rules and models are difficult to fully capture these changes. The reinforcement learning model of the present invention can process high-dimensional state space through deep neural networks to learn complex traffic patterns and decision-making strategies. The system can learn from historical data and continuously update its strategies from actual operations to adapt to new traffic conditions. Especially in the face of emergencies (such as traffic accidents, road construction, etc.), the system can quickly adjust the signal light control strategy to ensure smooth and safe traffic. In addition, by introducing technologies such as experience replay and target network, the system can maintain a stable learning process under different traffic conditions to avoid policy instability problems caused by data fluctuations. This improvement in adaptability and robustness makes the present invention perform better in dealing with complex and changing traffic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the system structure of an urban traffic optimization and planning system based on reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Example 1: Reference Figure 1, an urban traffic optimization and planning system based on reinforcement learning: the system includes: a traffic data acquisition unit, a reinforcement learning processing unit and a traffic planning unit; the traffic data acquisition unit is used to regard the target city traffic as a network, each intersection as a node, obtain the traffic data of the target city traffic on the road section between each two nodes, and assign a weight to each road section, define the traffic network state of each road section as a multidimensional tensor, and construct a nonlinear traffic flow dynamics model to update the multidimensional tensor at each time step; the reinforcement learning processing unit is used to use the signal timing matrix of the traffic lights on the road section between each two nodes of the target city traffic as the state space, define the target reward function of the reinforcement learning model, and calculate the corresponding action space when the target reward function is the maximum value through the pre-trained reinforcement learning model under the state space corresponding to each time step; the traffic planning unit is used to control the operation of the traffic lights on each road section according to the action space of each road section to complete the planning of the target city traffic.

[0045] Specifically, the traffic data acquisition unit first regards the traffic system of the target city as a complex network structure, in which each traffic intersection is regarded as a node, and each road section connecting these nodes is regarded as an edge. This networked way of thinking not only helps to analyze traffic flow in a structured manner, but also can accurately locate each traffic bottleneck and its impact on the overall traffic flow. When constructing the traffic network model, the traffic data acquisition unit will conduct comprehensive data collection on the road section between each two nodes, including but not limited to multi-dimensional information such as vehicle flow, average speed, traffic density and accident frequency. In order to ensure the timeliness and accuracy of the data, the unit usually integrates multiple sensing technologies and data sources, such as overhead cameras, ground induction coils, GPS data, and real-time traffic data from mobile applications. After standardization, these data are converted into a numerical format that can be further processed in the machine learning model. In order to reflect the importance of different road sections in the traffic network and their contribution to traffic flow, the system also assigns a weight to each road section. This weight is usually determined based on historical traffic data and the location and connection of the road section in the traffic network. For example, a road section connecting multiple main traffic arteries will have a relatively high weight because it plays a key role in the conduction and distribution of traffic flow. By setting these weights reasonably, the traffic data acquisition unit can more accurately reflect the actual traffic conditions, especially in special cases such as sudden traffic events or peak hours. More importantly, based on these basic data, the traffic data acquisition unit constructs a nonlinear traffic flow dynamics model. The introduction of this model is to more accurately simulate and predict the dynamic changes of traffic flow. Due to the complexity and unpredictability of urban traffic flow, simple linear models cannot meet the needs. Therefore, the traffic data acquisition unit adopts a nonlinear model to capture the complex relationships and dynamic characteristics in traffic flow. This model not only considers the mutual influence between various road sections, but also reflects the influence of external factors (such as weather conditions, special events, etc.) on traffic flow. By introducing a nonlinear model, the system can update the state of the traffic network at each time step to better reflect the real-time traffic conditions. During the update process of each time step, the traffic data acquisition unit uses a multidimensional tensor to represent the state of the entire urban traffic network. This multidimensional tensor is a high-dimensional data structure, each dimension of which represents a different traffic characteristic parameter. For example, a simple three-dimensional tensor may represent features such as flow, speed, and delay time. At each time step, the unit updates each element in the tensor based on the real-time traffic data, thereby dynamically capturing the instantaneous state of urban traffic. The use of this multi-dimensional tensor not only improves data processing efficiency, but also provides high-quality input data for subsequent reinforcement learning models.Especially when traffic conditions change dramatically, the use of this real-time updated tensor data can respond and adjust the signal light control strategy more quickly, achieving better traffic optimization effects. In addition, the traffic data acquisition unit must also consider the integrity and continuity of the data during data collection and processing. This means that when performing data modeling and analysis, the impact of data loss and errors must be minimized. Therefore, the system usually adopts a variety of redundant data collection methods to ensure the diversity and stability of data sources. For example, when the image data captured by the camera is blurred due to weather reasons, the data from the ground induction coil can be used as a supplement to ensure the integrity of the overall data. In order to improve the quality of the data, the traffic data acquisition unit will also apply some data cleaning and preprocessing techniques to filter out noise and outliers, making the input data more reliable and accurate.

[0046] In this system, each intersection and its signal light status in the traffic network are regarded as an independent intelligent agent. The reinforcement learning processing unit trains these intelligent agents to make optimal decisions in an uncertain and dynamically changing traffic environment. Specifically, the signal light status of each intersection, including the duration of green, red and yellow lights, is defined as part of the state space. The system updates the state of each intersection through the data collected in real time by the traffic data acquisition unit to form a global state representation at the current moment. At this time, the reinforcement learning processing unit needs to select an optimal signal light control action based on a pre-defined reward function in this state to optimize the efficiency of the entire urban traffic network. This definition of the state space allows the system to capture the complex characteristics of urban traffic and enables the control strategy of each signal light to adapt to changes in the current traffic conditions. In the reinforcement learning model, the action space refers to all possible signal light control strategies that the system can adopt. For each intersection, the reinforcement learning processing unit needs to decide how to set the time ratio of the signal light under a given traffic state, such as the duration of the green light, the waiting time of the red light, and the transition time of the yellow light. The combination of these decision variables constitutes the action space of the system. The reinforcement learning processing unit learns the optimal signal light control strategy by combining exploration and exploitation. In the exploration phase, the system tries different control strategies to understand the impact of different strategies on traffic flow; in the exploitation phase, the system selects the optimal strategy based on historical experience to maximize the long-term accumulated rewards. The reward function plays a vital role in the operation of the reinforcement learning processing unit. Its design needs to take into account multiple factors, such as the average waiting time of vehicles, the number of vehicles passing through, and the smoothness of traffic flow. Specifically, the reward function is usually designed as a multi-objective optimization function to minimize the waiting time and congestion of vehicles in the traffic network while maximizing the overall traffic capacity of the traffic flow. In this way, the system can adaptively adjust the control strategy of the signal light to meet the optimization needs in different time periods and different traffic conditions. For example, during peak traffic hours, the reward function may focus more on reducing the waiting time of vehicles, while during low peak hours at night, it may focus more on the smoothness of traffic flow and energy conservation.

[0047] In actual operation, the traffic planning unit first needs to obtain the latest signal light control strategy from the reinforcement learning processing unit. This strategy is usually expressed in the form of a matrix, in which each element corresponds to the state of the signal light at an intersection, such as the duration and switching order of the red, green or yellow light. Based on these control instructions, the traffic planning unit adjusts the state of the signal light at each intersection in real time. This real-time signal light control mechanism can effectively respond to changes in traffic flow in different time periods and sudden traffic events, thereby ensuring that the urban traffic network can maintain efficient operation under various conditions. In this way, the system can adaptively adjust the timing strategy of the signal light and dynamically optimize the traffic flow. The execution process of the traffic planning unit also involves continuous monitoring and feedback of traffic data to ensure the effectiveness of the signal light control strategy. Specifically, the traffic planning unit collects traffic flow data at each intersection and road section in real time, including vehicle queue length, average speed, vehicle waiting time, etc. These data are not only used to evaluate the effectiveness of the current signal light control strategy, but also provide feedback information for the next step of strategy optimization. For example, if the waiting time of vehicles on a certain road section exceeds the preset threshold, the system will feed back this information to the reinforcement learning processing unit in order to adjust and optimize the next signal light control strategy. Through such a closed-loop feedback mechanism, the traffic planning unit can continuously correct and optimize the control scheme of the signal light, thereby realizing the dynamic optimization of urban traffic. In addition, the traffic planning unit also needs to consider a variety of complex practical constraints, including but not limited to traffic rules, pedestrian priority, and emergency vehicle traffic requirements. In practical applications, the system must not only ensure the traffic efficiency of normal vehicles, but also take into account the rapid passage of pedestrians crossing the street and emergency rescue vehicles. Therefore, the traffic planning unit needs to have a high degree of flexibility and adaptability so that it can quickly make appropriate decisions in various traffic scenarios. For example, when an emergency rescue vehicle is detected approaching a certain intersection, the system will temporarily adjust the signal light status of the intersection to give priority to emergency vehicles to provide green light passage. This flexible adaptation to multiple constraints enables the traffic planning unit to provide more intelligent and humane signal control strategies when dealing with complex urban traffic environments.

[0048] Example 2: Traffic data of road section (i, j) at time t at least includes: vehicle density Average vehicle speed Traffic flow and average travel time The multidimensional tensor is:

[0049]

[0050] Where N is the number of nodes; i and j are both subscript indices with positive integer values; Represents the field of real numbers.

[0051] Specifically, vehicle density It refers to the number of vehicles on a unit length road section (i, j) at time t. It directly reflects the degree of congestion on the road section. The higher the vehicle density, the more crowded the road section is and the lower the freedom of vehicle travel. Vehicle density is an important parameter in the traffic network because it not only affects the average speed of vehicles, but also affects the speed of vehicles. It also affects the stability and flow of the overall traffic flow. Average vehicle speed It refers to the average speed of vehicles on road section (i, j) at time t. This parameter is not only affected by road conditions and traffic control (such as traffic lights), but also by vehicle density. Directly related. Generally speaking, as vehicle density increases, the average speed of vehicles decreases, which is also a common manifestation of traffic congestion. You can understand the current traffic flow efficiency. Traffic flow It represents the number of vehicles passing through the road section (i, j) per unit time at time t. Traffic flow is usually defined as vehicle density and the average vehicle speed The product of Traffic flow is an important indicator to measure the efficiency of road use and a key reference parameter for traffic signal optimization. It can help determine if a road segment is saturated or if signal timing needs to be adjusted to optimize traffic flow distribution. It refers to the average time required for a vehicle to travel from the starting point to the end point of a road section (i, j). This parameter combines the influence of all the above factors and provides an overall evaluation of traffic conditions in the time dimension. It not only reflects the road's capacity, but also indirectly reflects the efficiency of traffic management. For a traffic planning system, reducing the average travel time is an important optimization goal. Multidimensional tensor It is the state expression of the entire traffic network at time t. This tensor covers the key traffic parameters of all road sections and can capture the dynamic evolution of the traffic network through continuous updates in time and space. t is a three-dimensional matrix with dimensions N×N×4, where N is the number of nodes in the network, representing the total number of all intersections in the city. Each road segment (i, j) corresponds to a four-dimensional vector in the tensor Each element in this vector represents the above four key traffic parameters. t It is an indispensable core data structure in the traffic optimization system. It provides basic input data for the reinforcement learning processing unit. By analyzing and processing this tensor, the system can understand and predict the evolution of traffic flow and provide data support for the optimization of traffic light control strategy. Under the reinforcement learning framework, the traffic optimization system continuously updates the tensor St , and then train and adjust the model to gradually optimize the global transportation network. t It can accurately reflect the spatiotemporal dynamics of the traffic network, so it plays a vital role in optimizing signal timing, alleviating traffic congestion, and improving road traffic efficiency.

[0052] Example 3: Set the initial density of each road segment (i, j) to and the initial velocity is Define the time step Δt, and for each time step t=0, Δt, 2Δt, ..., divide each road segment (i, j) into M ij equal-length sub-segments; for each sub-segment m=1, 2, ..., M ij , the vehicle density, average vehicle speed, traffic flow and average travel time are updated using a preset nonlinear traffic flow dynamics model; the nonlinear traffic flow dynamics model is expressed using the following formula:

[0053]

[0054] in, is the maximum vehicle density of road section (i, j); is the maximum traffic flow of road section (i, j); is the free flow speed of the road section (i, j), which represents the average speed of vehicles when the road is unobstructed; α ij , β ij and γ ij are all characteristic index parameters of the preset road section (i, j); D ij is the diffusion coefficient of the road section (i, j); is the entrance flow of section (i, j) at time t; is the exit flow of section (i, j) at time t.

[0055] Specifically, consider describing vehicle density The partial differential equation that changes with time. The equation consists of two main parts, one is the characterization of the nonlinear changes in traffic flow, and the other is the consideration of the difference in import and export flows. The nonlinear changes in traffic flow are reflected in the first term of the equation, which describes the product of density and speed, multiplied by a nonlinear correction factor, which is here, represents the maximum vehicle density of the road section, α ij and β ij is an exponential parameter that controls the nonlinear characteristics. As the density increases, the correction factor gradually decreases, reflecting the saturation of traffic flow when it approaches the maximum density, which is a common congestion feature in urban traffic. Therefore, this part of the equation effectively describes the self-resistance characteristics of traffic flow: when the vehicle density is low, the traffic flow is high, and when the density is close to the maximum value, the flow rate decreases significantly, eventually leading to paralysis or severe congestion of the traffic network. At the same time, the equation also includes a term involving the difference in import and export flows, which partially reflects the impact of vehicle inflow and outflow flows on density within different time steps Δt, that is, This difference will affect the change in vehicle density on the road section. If the exit flow is greater than the entrance flow, the vehicle density will decrease; otherwise, the density will increase. This term not only takes into account the directionality of traffic flow, but also reflects the interdependence between different sections, which is particularly important in actual traffic networks. For example, when a key section is congested, the traffic conditions of surrounding sections will also be affected. This impact is mathematically expressed and quantified through the difference in inlet and outlet flow. Next, describe the average vehicle speed The time-varying partial differential equations also demonstrate the complexity and nonlinearity of traffic flow. The first part of the equation Describes the recovery characteristics of velocity. Free Flow Speed It refers to the maximum average speed that a vehicle can reach when the road is unobstructed. This item describes how the vehicle speed gradually recovers to the free flow speed from the current state, and is also affected by the current traffic flow. As the flow rate increases, the speed recovery rate slows down, and this relationship is determined by the parameter γ ij In this way, the equation can describe the dynamic characteristics of speed changes under different traffic conditions. In particular, in high-flow conditions, vehicle speed will be significantly suppressed, reflecting the deceleration phenomenon under congestion conditions. At the same time, the second part of the speed equation The dynamic relationship between speed and density is further characterized. Vehicle speed is negatively correlated with density, which means that when density increases, vehicle speed decreases. This is because the spacing between vehicles decreases and drivers need to slow down and brake frequently, thereby reducing the average speed. This description is particularly important because it reflects the following characteristics and interactions in actual driving behavior and provides a more detailed description of the microscopic behavior of traffic flow. In addition, the last term in the equation represents the effect of gravity and time delay on speed. This term specifically considers the effect of gravity on vehicle acceleration under complex terrain conditions such as uphill or downhill, as well as the inertial effect of speed change in some cases. By introducing this term, the model can better simulate the traffic flow characteristics under actual road conditions, and improve the adaptability to complex scenarios and prediction accuracy. This traffic flow dynamics model based on nonlinear partial differential equations has significant advantages over traditional linear models. First, it can more accurately reflect the nonlinear characteristics of actual traffic flow, such as the formation and propagation of congestion, and the interaction between vehicles. Second, it can flexibly adapt to various complex traffic conditions and environmental changes, such as emergencies and traffic accidents, and optimize the signal control strategy through adaptive adjustment of the model. Finally, by combining with the reinforcement learning algorithm, this nonlinear model can continuously learn and improve in a dynamically changing traffic environment, thereby continuously improving the intelligence level of the urban traffic optimization and planning system. Ultimately, the system can achieve refined management and optimization of urban traffic through intelligent optimization of signal light control in a highly dynamic and uncertain traffic environment, effectively improve the overall traffic efficiency of urban traffic, reduce congestion, and improve road safety.

[0056] Embodiment 4: The vehicle density, average vehicle speed, traffic flow and average travel time are updated by the following formula:

[0057]

[0058]

[0059] in, For the time t k+1 vehicle density; For the time t k+1 The average speed of vehicles; For the time t k+1 traffic flow; For the time t k+1 Average travel time; For the time t k vehicle density; For the time t k The average speed of vehicles; L ij is the length of the road segment (i, j).

[0060] Specifically, the updated formula for vehicle density is It captures the change in the number of vehicles on the road section per unit time. This change depends not only on the inflow and outflow of vehicles, but also on the vehicle density state of the previous time step. By calculating the rate of change of density The system can predict the density status at future moments. The advantage of this method is that it can dynamically reflect the accumulation or dispersion of traffic flow, thus providing a reference for the real-time optimization of traffic lights. For example, when the density of a certain section of road continues to increase, the system can timely adjust the traffic light duration of that section, increase the green light time to quickly divert traffic and avoid congestion. Average speed update It reflects the acceleration or deceleration process of the vehicle under different traffic conditions. Speed ​​change is not only a property of the vehicle itself, but also affected by the road conditions, vehicle density and external environment. Through this formula, the system can accurately simulate the dynamic changes of vehicle speed under given density conditions. For example, when the traffic flow density is high, the average speed tends to decrease. The system adjusts the signal light strategy to balance the traffic flow of each section, thereby improving the overall traffic efficiency. Traffic flow update formula The direct relationship between traffic flow, density and speed is revealed concisely. Traffic flow is a key indicator for measuring road capacity, which directly affects the timing strategy of traffic lights. In Example 4, the system can quickly evaluate the traffic conditions of the current road section by calculating traffic flow in real time and make corresponding control adjustments. For example, when the traffic flow exceeds a certain threshold, the system may need to extend the green light time or reduce the traffic flow of adjacent sections to prevent excessive congestion. Calculation of average travel time It further provides an evaluation criterion in the time dimension. It reflects the time required for a vehicle to pass a specific road section and is an important indicator for evaluating road traffic efficiency and driver experience. By updating the average travel time in real time, the system can dynamically perceive the traffic conditions of each road section. When the travel time increases significantly, it often indicates possible traffic congestion or other abnormal conditions. At this time, the system can adjust the traffic light strategy through the reinforcement learning algorithm to optimize the vehicle passing time and improve the overall traffic efficiency.

[0061] Example 5: The state space is a multidimensional tensor:

[0062]

[0063] Where K is the maximum number of phases at each intersection; is the green light time vector of node i at time t; is the phase order vector of node i at time t; is the offset and switching time vector of node i at time t;

[0064]

[0065] in is the green time of the sth phase of node i at time t;

[0066]

[0067] in represents the phase number of the sth execution sequence of node i at time t;

[0068]

[0069] in is the offset of node i relative to the system clock at time t; is the yellow light time of node i in the kth phase at time t; is the all-red time of node i at time t.

[0070] Specifically, the state space A t is a multidimensional tensor with dimension R N×(3K+2) , where N is the number of nodes in the network, usually representing the intersections where traffic lights are located, and K is the maximum number of phases at each intersection. With this setup, the system can represent the traffic signal control parameters of each node as a set of vectors, which include the green light time vector Phase Sequence Vector and offset and switching time vector This fully reflects the control status of each node at different time steps. is a key component of the system, which records the green light duration of each phase of node i at time t. This vector is expressed as:

[0071]

[0072] in Represents the green light time of the sth phase. The green light time directly affects the traffic efficiency. The duration of green lights in different phases determines the travel time of vehicles in each direction. By optimizing the green light time, the system can balance the traffic flow in different directions, reduce congestion and improve traffic efficiency. Phase order vector It describes the phase execution order of node i at time t. The sequence number of each phase The execution order of the phases in the current cycle is specified, such as left turn, straight ahead, right turn, etc. This order is crucial to the overall fluidity of traffic flow. A reasonable phase order can reduce vehicle waiting time and avoid traffic conflicts at intersections, thereby improving the overall traffic efficiency of the road. Contains a series of time parameters of node i at time t. These parameters include the offset relative to the system clock Yellow light time for each phase And all red time Offset It is the adjustment of the signal light cycle relative to the global system time, allowing the signal light cycle of each intersection to be fine-tuned, thereby achieving coordinated control of traffic signals. It is the transition time when each phase switches from green to red, which has an important impact on the driver's reaction and traffic safety. It is the time interval when the traffic lights in all directions show red lights, which is used to clear the intersection and ensure safety. By integrating these parameters into the state space tensor A t In the state space modeling method based on multidimensional tensors, the system can fully capture the state and time configuration of the traffic lights at each intersection. By analyzing and processing this multidimensional state space, the reinforcement learning model can learn how to optimally adjust these parameters, thereby optimizing the control strategy of traffic lights under different traffic conditions. For example, the system can respond to real-time changes in traffic flow by dynamically adjusting the green light time and phase sequence to avoid long waiting times at intersections; it can also coordinate the signal cycles of adjacent intersections by adjusting the offset to achieve smooth flow of traffic. This state space modeling method based on multidimensional tensors enables the system to have higher flexibility and accuracy when dealing with complex urban traffic optimization tasks. It can not only reflect the current traffic conditions, but also keep the traffic signal control strategy in the optimal state through real-time updates and optimizations, reduce traffic congestion, improve road use efficiency, and ultimately achieve intelligent management and planning of urban traffic. This embodiment shows how to effectively use reinforcement learning to cope with challenges in urban traffic through detailed state space modeling, and provides important technical support for future intelligent transportation systems.

[0073] Example 6: Target reward function R(S) of reinforcement learning model t , A t ) is expressed using the following formula:

[0074] R(S t , A t )=w flow ·R flow (S t , A t )+w stability ·R stability (S t , A t );

[0075] Among them, R flow (S t , A t ) is the traffic efficiency objective reward function; R stability (S t , A t ) is the traffic stability objective reward function; w flowis the preset traffic efficiency weight; w stability is the preset traffic stability weight; where w flow +w stability =1.

[0076] Specifically, the traffic efficiency objective reward function R flow (S t , A t ) is designed to maximize traffic flow and minimize vehicle waiting time, thereby improving the capacity of the entire urban transportation network. In a complex urban traffic environment, the size of traffic flow and the average waiting time of vehicles directly reflect the efficiency of the transportation system. Efficient traffic management requires that as many vehicles as possible pass through each intersection smoothly without sacrificing traffic safety. Therefore, R flow (S t , A t ) usually considers multiple factors, such as the vehicle passing rate at each intersection, the length of the fleet, the traffic density of each road section, etc. By optimizing these parameters, the system can reduce the average waiting time of vehicles and increase the number of vehicles passing per unit time, thereby improving traffic efficiency. Traffic stability objective reward function R stability (S t , A t ) focuses on ensuring the smooth flow of traffic and the rationality of signal light switching. Stable traffic flow not only helps reduce the risk of traffic accidents, but also reduces vehicle fuel consumption and exhaust emissions. Frequent switching of signal lights will lead to frequent vehicle starts and stops, which not only increases fuel consumption, but may also cause driver discomfort and increase the chance of traffic accidents. Therefore, R stability (S t , A t ) usually considers factors such as the phase switching frequency of the signal light, the balance of the phase duration, and the signal coordination between adjacent intersections. By optimizing these factors, the system can reduce unnecessary signal light switching, ensure the smooth passage of vehicles, and thus improve the overall stability of traffic. Weight w flow and w stability The choice of w directly affects the optimization direction of the reward function. The weights are usually set based on the current traffic conditions and management goals. For example, during rush hour, the main goal of a city may be to divert as much traffic as possible, so the system may set a higher w flow , to emphasize the optimization of traffic efficiency; at night or during off-peak hours, when traffic volume is low, city managers may pay more attention to traffic stability and environmental impact. stabilityThis method of dynamically adjusting weights enables the system to flexibly adjust the optimization target under different time periods and traffic conditions, thus achieving more refined traffic management. By combining the two key factors of traffic efficiency and stability, the target reward function R(S t , A t ) provides a comprehensive optimization framework for the reinforcement learning model. This framework can not only guide the system to make the best signal light control decision under different traffic conditions, but also enable the system to maintain flexibility and adaptability in a changing traffic environment. Through continuous learning and adjustment, the reinforcement learning model can gradually master the optimal control strategy under various complex traffic conditions, thereby maximizing traffic flow and optimizing road use. Specifically, when the system faces high-density traffic flow, the R in the reward function flow (S t , A t ) will dominate the optimization process, and the system will give priority to those signal light timing schemes that can maximize traffic flow and minimize waiting time. At this time, the signal light may be set to a longer green time, especially for the direction with heavy traffic, to ensure that traffic can pass through the intersection quickly and reduce the bottleneck effect. Under the condition of relatively stable traffic flow, R in the reward function stability (S t , A t ) will play a greater role, and the system will select strategies that can reduce the frequent switching of traffic lights and maintain stable traffic flow. This will not only help reduce fuel consumption and emissions, but also improve driving experience and driving safety. Through this multi-objective optimization design, Example 6 shows how to effectively optimize the control strategy of urban traffic lights through a carefully designed reward function under the reinforcement learning framework. The advantage of this method is that it can dynamically balance the two goals of traffic efficiency and stability, and make intelligent adjustments based on actual traffic conditions. This intelligent traffic management strategy significantly improves the capacity of urban traffic, reduces the possibility of congestion and accidents, and provides an important technical support for the construction and development of smart cities.

[0077] Example 7: The traffic efficiency objective reward function is expressed using the following formula:

[0078]

[0079] in, is the total green light time of road section (i, j) at time t; is the optimal vehicle density of the preset road section (i, j); σ ij is the density tolerance parameter of the preset road section (i, j); is the most critical speed of the road section (i, j), which means the average speed corresponding to the road section (i, j) when the maximum traffic flow of the road section (i, j) is lower than the preset traffic critical value; is the maximum allowed speed of road section (i, j); The signal offset of the road section (i, j) at time t; |·| is the operation of finding the modulus of the vector.

[0080] Specifically, the structure of the function consists of multiple multiplied factors, each of which represents a different aspect that affects traffic efficiency. First, The ratio reflects the relationship between traffic flow and average travel time. The larger the ratio, the more vehicles pass through the road section in a unit of time, and the higher the traffic efficiency. is the vehicle density and average speed Therefore, it not only reflects the congestion level of the current road section, but also describes the speed of vehicles. It directly affects the driver's travel experience and is an important indicator for optimizing traffic efficiency. By maximizing this ratio, the system can optimize those sections with high traffic and low congestion, thereby improving the traffic efficiency of the overall traffic network. In addition, the complexity of the signal light control strategy is introduced as an important adjustment factor. Indicates that and are the moduli of the phase sequence vector and the green light time vector, respectively. The introduction of this factor is intended to encourage the system to choose a simpler signal light configuration scheme to reduce the disturbance of overly complex phase switching to traffic flow. Frequent signal switching and complex phase arrangements often lead to frequent vehicle starts and stops, increased fuel consumption and exhaust emissions, and may also cause more traffic accidents. By limiting the complexity of the control strategy, the system can achieve coordinated control of traffic lights and reduce unnecessary switching frequency, thereby improving the smoothness of traffic flow and the efficiency of vehicle traffic. At the same time, the Gaussian distribution term in the reward function A preference is provided for the optimal vehicle density. Vehicle Density It is an important parameter to measure traffic mobility, and its optimal value It is usually the result of a balance between traffic flow and road utilization. The use of the Gaussian distribution function ensures that when the vehicle density is close to the optimal value, the system will receive a higher reward, thereby encouraging traffic flow to remain in an efficient but not overly congested state. If the vehicle density deviates from the optimal value, whether it is too high or too low, the reward will decrease, which means that the system needs to adjust the signal light strategy to re-optimize the traffic flow distribution. This density control strategy can effectively prevent traffic flow from being over-concentrated or over-dispersed, improve the utilization of road resources, and avoid traffic congestion or road vacancy in extreme cases. In addition, the reward function also takes into account the impact of vehicle speed on traffic efficiency. In this item, the system can be optimized for different speed conditions. is the average speed of vehicles on the road section, the critical speed is the speed that maximizes traffic flow. This item uses the nonlinear characteristics of the hyperbolic tangent function (tanh) to increase the value of the reward function when the speed approaches the critical speed, thereby encouraging the system to keep the vehicle speed close to the critical speed when optimizing the signal control strategy. This design ensures that vehicles pass through the road section at the speed closest to the optimal speed, improving the efficiency of traffic flow. If the speed is too low or too high, the reward will be reduced accordingly, indicating that the system's current signal control strategy may need to be adjusted. Finally, the impact of the green light time and signal offset on the reward function is expressed by Here, is the sum of the green light time of the road section (i, j), reflecting the length of time a vehicle can pass through the road section; and is the signal offset, which indicates the degree of offset of the signal cycle. The logarithmic function of the green time helps to smooth the impact of the signal time on the reward and avoid system instability caused by large changes in the green time. At the same time, the inverse square root of the signal offset reflects the requirement for signal time synchronization. A large offset may lead to uncoordinated signal cycles, thus affecting the fluidity of the entire traffic network. Therefore, by balancing the green time and signal offset, the system can better coordinate the signal configuration between different sections and intersections, and improve the continuity and efficiency of traffic flow.

[0081] Example 8: The traffic stability objective reward function is expressed using the following formula:

[0082]

[0083] Among them, w ij is the weight of the preset road section (i, j); is the average delay of link (i, j) at time t; is the entropy of all output flows of node i; H maxis the maximum possible entropy of all output flows of node i when the distribution is completely uniform; is the queue length of section (i, j) at time t; is the target total queue length of node i; σ l It is the preset queue length tolerance parameter.

[0084] Specifically, the first factor in the function It is used to measure the average delay of node i. Here, w ij is the preset weight of the road segment (i, j), reflecting the importance or priority of the road segment, and is the average delay time of the road section at time t. By weighted average, the system can comprehensively evaluate the delays of different sections to ensure that key sections receive more attention during traffic optimization. Delay time is an important indicator of poor traffic flow. Large delays mean that vehicles may have to wait for a long time on certain sections. The system improves the overall traffic flow by minimizing this indicator. Next, the second factor of the function The entropy H of the output traffic is introduced to measure the uniform distribution of traffic. Entropy is used in information theory to measure uncertainty or the uniformity of distribution. In traffic management, a higher entropy value indicates that the traffic is more evenly distributed among multiple exit sections. By introducing this factor, the system encourages a more balanced distribution among all the output flows of node i, avoiding excessive congestion in some sections while other sections are relatively empty. The maximum entropy value H max It represents the traffic entropy when the distribution is completely uniform. Therefore, the closer the value of this factor is to 0, the more uneven the traffic distribution is. By optimizing the entropy value, the system can improve the balance of traffic flow and make the overall traffic network more stable. The third factor The Gaussian distribution function is used to reflect the deviation between the queue length and the target length. is the number of vehicles waiting in line on the road section (i, j) at time t, and is the target total queue length of node i. The system ensures that the traffic signal control strategy can maintain the queue length within a reasonable range by minimizing the deviation between the actual queue length and the target value, avoiding traffic delays caused by excessively long queues. The preset queue length tolerance parameter σ l The system's tolerance for this deviation is defined. When the queue length approaches the target value, the value of the reward function will increase, and vice versa. This design helps to prevent congestion caused by excessive queue length while optimizing traffic flow, thereby maintaining the continuity of traffic flow. In addition, the function also takes into account the balance of the green light time of the traffic light and the coordination of the phase timing. By introducing With this factor, the system emphasizes the balance of the green light time of traffic lights in different phases. A higher green light time balance indicates that traffic flows in different directions are more fair in time allocation, which helps to avoid excessive backlog of vehicles in certain directions. On the contrary, excessively unbalanced green light time allocation may cause traffic flows in some directions to be unable to be effectively diverted, thereby increasing the instability of the overall traffic system. By optimizing this ratio, the system can achieve a better balance in signal light time allocation, thereby improving the stability of traffic flow. Finally, the factor in the function It further reflects the relationship between the phase sequence and the green light time and the signal offset. and are the phase sequence vector and the green light time vector, respectively, and is the signal offset. This factor is intended to encourage the reasonable configuration of phase order and green light time in time and space, and to prevent the problem of timing incoordination caused by excessive signal offset. A large signal offset may cause the signal lights between adjacent intersections to be out of sync, which in turn causes discontinuity and congestion in traffic flow. Therefore, by minimizing the signal offset and optimizing the phase order, the system can achieve smoother traffic signal control, reduce unnecessary pauses and starts, and improve the stability and efficiency of overall traffic. The traffic stability target reward function in Example 8 provides strong mathematical support for the optimization of traffic signal control strategies through comprehensive considerations in multiple dimensions. By accurately measuring key factors such as delays, flow distribution, queue length, and signal light time configuration, the system can optimize the control strategy of signal lights while ensuring the stability of traffic flow, and ultimately achieve intelligent management of urban traffic. This multi-factor combination design not only improves the stability of traffic flow, but also reduces potential traffic fluctuations and imbalances, laying a solid foundation for building an efficient and stable urban transportation network.

[0085] Example 9: The parameter θ update formula of the reinforcement learning model is:

[0086]

[0087] Among them, η is the learning rate, c is the gradient clipping threshold, λ is the weight decay coefficient, λ1 is the L2 regularization coefficient; θ new is the updated reinforcement learning parameter; L(θ) is the loss function of reinforcement learning; clip(, -c, c) is the gradient clipping function, which limits the gradient value to the range of [-c, c], and c is the preset value; is the gradient operator; Represents a replay buffer of experiences learned from reinforcement learning Sampling conversion (S t , A t , R(S t , A t ), At ′) take the average, A t ' is the next action; represents the square of the L2 norm; θ - is the target parameter of reinforcement learning; Q(S t , A t ′;θ - ) is a given state S t and action A t ′, using parameter θ - The predicted Q value.

[0088] Specifically, the core formula for parameter update is The structure of this formula shows how the model updates from the current parameters θ to the new parameters θ at each iteration new The update process mainly relies on the gradient descent method, which is a widely used optimization method in deep learning and reinforcement learning. By calculating the gradient of the loss function L(θ) with respect to the parameter θ The model can adjust parameters in the opposite direction of the gradient to minimize the loss function. The purpose of this process is to make the model's predictions closer and closer to the actual optimal strategy, thereby improving the effectiveness of the traffic light control strategy in different traffic scenarios. However, the simple gradient descent method may face some challenges in practical applications, especially in the training of deep neural networks, where gradient explosion or vanishing phenomena can lead to instability in model training. These problems are particularly prominent in complex multi-state systems such as traffic optimization. In order to alleviate these problems, the gradient clipping technology is introduced into the formula. Limit the gradient value to the range of [-c, c]. The purpose of gradient clipping is to prevent the gradient update from being too large or too small, which will lead to instability in model training. When the absolute value of the gradient exceeds the preset threshold c, the clipping function will limit it to c or -c, thereby avoiding the large or small jump of parameter updates caused by excessive gradients, thereby ensuring the training stability and convergence of the model. Gradient clipping is particularly important in traffic optimization tasks because traffic state changes are complex and highly dynamic. Through effective gradient control, the model can learn and adapt to these changes more stably. Another important component of the update formula is the weight decay term -λθ. Weight decay is a regularization technique designed to prevent the model from overfitting. Overfitting means that the model performs well on the training data but performs poorly on new data because the model overfits the noise or features in the training data. Weight decay reduces the parameter value in each update so that the model does not rely too much on certain specific parameters, thereby improving its generalization ability on new data. This regularization is particularly critical in urban traffic optimization, as traffic flows and patterns are highly uncertain and variable in time and space. Through weight decay, the model can better adapt to different traffic conditions and changes, maintaining the robustness of the signal control strategy. The loss function L(θ) is the key to the parameter update process, which defines the optimization objective. Specifically, It contains two parts: one is the mean square error between the predicted Q value and the target Q value, and the other is the L2 regularization term. The mean square error term (yQ(S t , A′ t ;θ)) 2 It reflects the gap between the model's prediction and reality. Here, It is the target Q value based on the Bellman equation. The Bellman equation is one of the core equations in reinforcement learning, which defines the value of the state-action pair with the goal of maximizing the long-term cumulative reward. By minimizing this mean square error, the model continuously adjusts the parameters so that the predicted Q value gradually approaches the actual optimal Q value. This self-improvement mechanism enables the system to effectively learn the optimal traffic light control strategy under different traffic conditions. In the framework of reinforcement learning, this update mechanism is called Q-learning, especially deep Q-learning, when the Q value is estimated by a neural network. In urban traffic optimization, the advantage of Q-learning is that it can handle high-dimensional state spaces, such as a variety of different traffic flow patterns and signal light configurations. The introduction of the experience replay buffer D further enhances the stability of the learning process. By randomly sampling from the experience replay buffer, This method breaks the correlation between samples, making each update based on an independent set of data, which can effectively reduce the variance of the model and improve the efficiency and stability of learning. Experience replay can ensure that the model learns more comprehensive traffic flow characteristics and improve its adaptability when facing diverse traffic conditions. Another key part is the L2 regularization term in the loss function. L2 regularization prevents overfitting by adding the square of the parameter norm to the loss function to prevent the parameter from being too large. For traffic signal control tasks, the optimal solution for traffic light configuration is often not a simple fixed strategy, but needs to be adjusted dynamically according to the current traffic flow. Therefore, excessive complexity of model parameters will make it difficult to maintain stable performance under different traffic conditions. By adding the L2 regularization term, the parameter value of the model is limited to a reasonable range, thereby enhancing its generalization ability in new environments. In addition, the target parameter θ in the formula - Separating the current parameters θ is also a common technique in reinforcement learning. - It is copied from the current parameter θ after a specific time interval, rather than being updated at each step. This method is introduced to avoid the instability caused by the rapid change of the target Q value. Since the target parameter is updated less frequently, this design can reduce the variance of the predicted target and improve the stability and convergence of the model. For urban traffic optimization tasks, this stability is particularly important because the changes in the signal light control strategy should not be too frequent and drastic to avoid sudden fluctuations and congestion in traffic. The learning rate η in the formula also plays a vital role. The learning rate controls the magnitude of each parameter update. In practical applications, the learning rate must be selected with caution. If the learning rate is too high, the model update step is too large, which may lead to skipping the global optimal solution or oscillating back and forth on the loss surface; if the learning rate is too low, the update step is too small, which will lead to slow convergence and long training time. To solve this problem, some learning rate decay strategies are usually used to gradually reduce the learning rate as the training progresses, so as to conduct a fast search in the early stage and make detailed adjustments when approaching the optimal solution. Such a strategy is particularly effective in traffic optimization, because the initial stage requires quickly finding a better signal light control strategy framework, while the subsequent stages require fine-tuning the strategy to adapt to more complex traffic flow dynamics.

[0089] The reinforcement learning model parameter update formula in Example 9 demonstrates a comprehensive optimization mechanism, which not only takes into account the stability and convergence of the model during the training process, but also enhances the robustness of the model in complex dynamic environments through a variety of regularization techniques. This method enables the urban traffic optimization system to continuously learn and adjust the signal light control strategy to cope with different traffic modes and flow changes, and ultimately achieve more efficient and stable traffic management. This update formula theoretically provides a solid foundation for the application of deep reinforcement learning in traffic optimization, and demonstrates its strong adaptability and optimization capabilities in practice, making important contributions to the intelligence and modernization of urban traffic management. Through such a reinforcement learning strategy, the system can continuously improve its decision-making level, reduce traffic congestion, and improve the overall operating efficiency and road safety of urban traffic in a highly dynamic and uncertain traffic environment.

[0090] The present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. Urban traffic optimization and planning system based on reinforcement learning, characterized by: The system includes: a traffic data acquisition unit, a reinforcement learning processing unit and a traffic planning unit; the traffic data acquisition unit is used to regard the target city traffic as a network, each intersection as a node, obtain the traffic data of the target city traffic on the road section between each two nodes, assign a weight to each road section, define the traffic network state of each road section as a multidimensional tensor, and construct a nonlinear traffic flow dynamics model to update the multidimensional tensor at each time step; the reinforcement learning processing unit is used to use the signal timing matrix of the traffic lights on the road section between each two nodes of the target city traffic as the state space, define the target reward function of the reinforcement learning model, and calculate the corresponding action space when the target reward function is the maximum value through the pre-trained reinforcement learning model under the state space corresponding to each time step; the traffic planning unit is used to control the operation of the traffic lights on each road section according to the action space of each road section to complete the planning of the target city traffic; The target reward function R(S) of the reinforcement learning model t , A t ) is expressed using the following formula: R(S t ,A t )=w flow ·R flow (S t ,A t )+w stability ·R stability (S t ,A t ); Among them, R flow (S t , A t ) is the traffic efficiency objective reward function; R stability (S t , A t ) is the traffic stability objective reward function; w flow is the preset traffic efficiency weight; w stability is the preset traffic stability weight; where w flow +w stability =1; A t is the state space; S t is a multidimensional tensor; The traffic data of road section (i, j) at time t includes at least: vehicle density Average vehicle speed Traffic flow and average travel time The multidimensional tensor is: Where N is the number of nodes; i and j are both subscript indices with positive integer values; represents the field of real numbers; Where K is the maximum number of phases at each intersection; is the green light time vector of node i at time t; is the phase order vector of node i at time t; is the offset and switching time vector of node i at time t; in, is the sum of green light time of road section (i, j) at time t; is the optimal vehicle density of the preset road section (i, j); σ ij is the density tolerance parameter of the preset road section (i, j); is the most critical speed of the road section (i, j), which means the average speed corresponding to the road section (i, j) when the maximum traffic flow of the road section (i, j) is lower than the preset traffic critical value; is the maximum allowed speed of road section (i, j); The signal offset of the road section (i, j) at time t; |·| is the operation of finding the modulus of the vector; Among them, w ij is the weight of the preset road section (i, j); is the average delay of link (i, j) at time t; is the entropy of all output flows of node i; H max is the maximum possible entropy of all output flows of node i when the distribution is completely uniform; is the queue length of section (i, j) at time t; is the target total queue length of node i; σ l It is the preset queue length tolerance parameter.

2. The urban traffic optimization and planning system based on reinforcement learning as claimed in claim 1, characterized in that: Set the initial density of each road segment (i, j) to and the initial velocity is Define the time step Δt, and for each time step t=0, Δt, 2Δt, ..., divide each road segment (i, j) into M ij equal-length sub-segments; for each sub-segment m=1, 2, ..., M ij , the vehicle density, average vehicle speed, traffic flow and average travel time are updated using a preset nonlinear traffic flow dynamics model; the nonlinear traffic flow dynamics model is expressed using the following formula: in, is the maximum vehicle density of road section (i, j); is the maximum traffic flow of road section (i, j); is the free flow speed of the road section (i, j), which represents the average speed of vehicles when the road is unobstructed; α ij , β ij and γ ij are all characteristic index parameters of the preset road section (i, j); D ij is the diffusion coefficient of the road section (i, j); is the entrance flow of section (i, j) at time t; is the exit flow of section (i, j) at time t.

3. The urban traffic optimization and planning system based on reinforcement learning as claimed in claim 2, characterized in that: The vehicle density, average vehicle speed, traffic flow and average travel time are updated using the following formula: in, For the time t k+1 vehicle density; For the time t k+1 The average speed of vehicles; For the time t k+1 traffic flow; For the time t k+1 Average travel time; For the time t k vehicle density; For the time t k The average speed of vehicles; L ij is the length of the road segment (i, j).

4. The urban traffic optimization and planning system based on reinforcement learning as claimed in claim 3, characterized in that: in is the green time of the sth phase of node i at time t; in represents the phase number of the sth execution sequence of node i at time t; in is the offset of node i relative to the system clock at time t; is the yellow light time of node i in the kth phase at time t; is the all-red time of node i at time t.

5. The urban traffic optimization and planning system based on reinforcement learning as claimed in claim 1, characterized in that: The parameter θ update formula of the reinforcement learning model is: Among them, η is the learning rate, c is the gradient clipping threshold, λ is the weight decay coefficient, λ1 is the L2 regularization coefficient; θ new is the updated reinforcement learning parameter; L(θ) is the loss function of reinforcement learning; clip(·, -c, c) is the gradient clipping function, which limits the gradient value to the range of [-c, c], and c is the preset value; is the gradient operator; Represents a replay buffer of experiences learned from reinforcement learning Sampling conversion (S t , A t , R(S t , A t ), A t ′) take the average, A t ' is the next action; represents the square of the L2 norm; θ - is the target parameter of reinforcement learning; Q(S t , A t ′;θ - ) is a given state S t and action A t ′, using parameter θ - The predicted Q value.

Images