Expressway multi-bottleneck layered ramp coordination control method based on deep reinforcement learning

Abstract

The invention provides an expressway multi-bottleneck layered ramp coordination control method based on deep reinforcement learning so as to improve the overall traffic efficiency of a system. According to the method, an expressway comprising a plurality of bottleneck road sections is taken as a whole, and a ramp coordination control method is constructed by adopting a deep reinforcement learning algorithm according to a macroscopic fundamental diagram theory. According to the method, a hierarchical control framework is adopted, wherein upper-layer control is combined with an expressway macroscopic fundamental diagram theory and a deep reinforcement learning algorithm, and expected ramp total flow entering a main line is output; and a refined local control strategy is adopted in lower-layer control, the total flow of the ramps is distributed to all the ramps, and the expected flow of all the ramps is obtained. The control method provided by the invention not only can play the advantages of low cost and high robustness based on macroscopic fundamental diagram control, but also has the advantages of being independent of an accurate model and capable of adapting to an uncertain environment based on deep reinforcement learning control, so that the requirement of efficient ramp coordination control of a large-range expressway can be effectively met.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology for expressways, including highways and expressways, and particularly to a multi-bottleneck hierarchical ramp coordination control method for expressways based on deep reinforcement learning. Background Technology

[0002] Expressways, including highways and expressways, are characterized by independent right-of-way, high traffic volume, and rapid transit, forming the backbone of the road traffic system. Within urban areas, ramps are densely distributed, and traffic demand is concentrated during peak hours, easily creating multiple traffic bottlenecks. These bottlenecks interact significantly: when traffic demand is high, implementing ramp control on a single bottleneck, even if it avoids congestion at that bottleneck, will not effectively improve the overall outbound traffic flow if other downstream bottlenecks remain congested; changes in traffic conditions at one bottleneck caused by control can trigger a chain reaction of changes in traffic conditions at surrounding bottlenecks.

[0003] For expressways with multiple bottlenecks, existing methods for ramp coordination control are mainly based on basic road segment maps and can be divided into three categories. One category is simple and easy-to-implement control methods based on road segment density thresholds, such as single-point ramp feedback control based solely on the local density at the merging bottleneck. These methods are simple to implement but do not adequately consider the overall traffic efficiency of the road network and struggle to achieve global coordination among multiple bottlenecks. Another category is optimization control methods based on traffic models, using cellular transfer models (CTM) or METANETs as prediction models to solve for the optimal ramp flow control sequence within a certain time domain. This type of method can avoid the short-sighted behavior of feedback control to some extent and theoretically achieves better overall control performance. However, it highly depends on accurate traffic flow prediction models; the error between the deterministic prediction model and actual stochastic traffic operation can be large, and the computational complexity is high, making real-time application difficult. A third category is model-free control methods based on reinforcement learning, which are generally designed based on single-point ramp control methods. This type of method can adapt well to the inherent uncertainties of the traffic environment and can generally solve problems in real time. However, its computational cost increases exponentially with the expansion of road scale, and it may even fail to converge, resulting in control failure.

[0004] Furthermore, traditional ramp coordination control methods often construct control strategies targeting single bottlenecks or a limited number of bottlenecks, failing to fully utilize the macroscopic fundamental map characteristics of multiple bottlenecks on expressways. Moreover, the optimization control methods within these methods often face the challenge of high computational costs, especially model-based optimization control methods which have limited adaptability to the stochasticity of traffic operations. Therefore, there is an urgent need for a method that can take multiple bottlenecks on expressways as the overall object, fully utilize the overall traffic evolution mechanism provided by the macroscopic fundamental map without requiring an accurate traffic model, and combine data-driven deep reinforcement learning methods to achieve hierarchical ramp coordination control.

[0005] The technical differences compared to existing technologies are as follows:

[0006] Technical Comparison with Comparative Patent 1 (CN112102628A / family CN112102628B, "Ramp Coordination Control and Guidance Method for Bottlenecks of Two-Deck Expressways")

[0007] Differences between application scenarios and technical goals;

[0008] In contrast to Patent 1, which explicitly addresses the "dual-layer highway bottleneck" scenario and alleviates bottleneck congestion by combining single-point ramp control with traffic guidance under dual-layer linkage conditions, and incorporates control algorithms and coordination mechanisms such as ALINEA and Bottleneck to achieve dual-layer linkage and traffic guidance dissemination, this patent focuses on long-distance, multi-bottleneck sections of expressways (typically 10-20 km long, with densely distributed bottlenecks). Its core objective is to solve problems such as "insufficient global coordination, strong dependence on model accuracy, and high computational complexity" under multi-bottleneck coupling conditions. It constructs a hierarchical ramp coordination control system based on a macroscopic basic graph and deep reinforcement learning to improve the overall system efficiency and robustness.

[0009] Differences in overall technical approach and control architecture;

[0010] The core approach of Patent 1 is to calculate ramp control quantities separately around the two-layer structure, and further achieve cross-layer allocation and guidance through "coordination + guidance," emphasizing the coordination and induced release of the two-layer system, with the control logic mainly based on existing feedback / rule-based algorithms. This patent, however, adopts a hierarchical control architecture: the upper layer takes the overall macroscopic state of the mainline and ramp aggregated information as input, directly outputting "the total ramp flow expected to enter the mainline in the current control cycle"; the lower layer, under total volume constraints, constructs weights based on the congestion level of each ramp corresponding to the mainline segment and allocates flow to obtain the flow of each ramp.

[0011] The core algorithm depends on the difference in the hierarchical role of the "learning module";

[0012] Compared to Patent 1, which uses classic control / coordination algorithms such as ALINEA and Bottleneck as its core and belongs to "model / rule-driven feedback control and coordination," this patent does not reflect a framework of policy learning and offline training followed by online inference using model-free reinforcement learning. In contrast, this patent models the upper-level "ramp total flow control" as a Markov decision process and uses PER-D3QN (or alternatively TD3, etc.) for offline training. In the online phase, only forward inference is needed to output the total ramp flow for the control cycle, emphasizing reduced reliance on accurate prediction models and improved adaptability to stochastic demand fluctuations.

[0013] Differences in global collaboration mechanisms across multiple bottlenecks;

[0014] Compared to Patent 1, which primarily serves cross-layer coordination and traffic guidance dissemination within a two-layer structure, focusing on "two-layer linkage" in its optimization targets and coordination dimensions, this patent's "coordination" addresses global coupling within long-distance, multi-bottleneck sections by implementing ramp coordinated control. It incorporates multiple bottlenecks into a unified state representation through MFD macro-state analysis. The upper layer uses system outflow as the primary reward, combined with key density threshold penalties to accelerate convergence. The lower layer uses congestion weights to achieve refined allocation of "more congested bottlenecks corresponding to more stringent flow restrictions on ramps," thereby achieving coordinated optimization among multiple bottlenecks.

[0015] Technical comparison with Patent 2 (CN112150813A, "A method for controlling entrance ramps of long-distance downstream bottleneck sections of highways");

[0016] Differences between research subjects and problem setting;

[0017] Compared to Patent 2, which focuses on scenarios where "there is a non-negligible time delay between the entrance ramp and a bottleneck located far downstream," the core issue is that traditional single-point ramp control strategies struggle to eliminate the reduced control effectiveness caused by this time delay. Therefore, it proposes a feedback control strategy based on traffic flow detectors and "fuzzy adaptive PID" to address the hysteresis. This patent, however, addresses the problem of multi-ramp coordinated control in "densely distributed bottlenecks on expressways." Bottleneck sources include merging, diverging, and geometric changes. It emphasizes global total flow control and local differentiated allocation under multi-bottleneck coupling, rather than establishing a dedicated PID / fuzzy compensation control around a single "long-distance downstream bottleneck time delay."

[0018] Differences between control strategy form and output variables;

[0019] In contrast, Patent 2 uses "ramp regulation rate r(k)" as the control variable and employs a typical feedback control structure: it constructs the control input based on the deviation between the desired density and the actual density, and adjusts the PID parameters online using fuzzy rules. This patent, however, is structurally divided into upper and lower layers: the upper layer's action is directly defined as "the total flow of the ramps expected to enter the main line in the current control cycle," and is output by a reinforcement learning strategy within the discrete action space; the lower layer then allocates the total flow to be restricted to each entrance ramp according to congestion weights and converts it into green light time for execution.

[0020] Differences in model and rule dependence and robustness path;

[0021] The key difference between Patent 2 and Patent 3 lies in the use of "fuzzy rules + PID" to compensate for time delay and nonlinearity, which is essentially still driven by controller structure and rule design. This patent, however, explicitly employs model-free deep reinforcement learning, learning a "macro-state-total-control" strategy through interaction with the simulation environment. This aims to reduce reliance on precise prediction models / control rules and maintain robust control performance under random demand fluctuations.

[0022] Differences in multi-ramp coordination and allocation mechanisms;

[0023] Compared to Patent 2, which focuses on a single control logic (fuzzy adaptive PID) to output the ramp regulation rate without highlighting a "differentiated flow restriction allocation mechanism among multiple entrance ramps," this patent explicitly constructs a congestion weight based on "weighted average density of the main line section + density distribution difference index," and allocates the total flow that needs to be restricted across ramps accordingly. This allows ramps with more severe congestion to receive stricter flow restrictions, achieving multi-ramp coordination.

[0024] Technical comparison with Patent 3 (CN117690291A, "Method, System, Equipment and Medium for Controlling Ramps in Merging Bottlenecks of Highways Based on Improved ALINEA")

[0025] Differences between the scope of controlled objects and the type of bottleneck;

[0026] Compared to Patent 3, which focuses on ramp control in "highway merging bottleneck areas," the scenario focuses on the control and execution of entrance ramps at typical merging bottlenecks. In contrast, this patent targets the collaborative control of "long-distance expressways with multiple bottleneck sections," where bottlenecks originate from a wider range of sources (merging, diversion, geometric changes, etc.) and multiple bottlenecks are coupled. The control objective is to achieve section-level global optimization through coordinated control of multiple ramps.

[0027] Core differences in control methods;

[0028] The method kernel of Comparative Patent 3 belongs to "feedback ramp control based on improved ALINEA", and generally still forms ramp control volume with ALINEA - like occupancy and density error feedback as the core. In this patent, the upper layer outputs "total ramp flow" with an endogenous learning DRL strategy, whose state is composed of MFD macroscopic indicators and ramp aggregation information, and the reward drives strategy learning with the system out - flow (combined with critical density threshold penalty); the lower layer realizes cross - ramp allocation with congestion weights and implements signal control, achieving a hierarchical closed - loop of "learning decision + executable control instructions".

[0029] Differences between multi - ramp coordination and the "total - quantity - allocation" mechanism;

[0030] Comparative Patent 3 forms a control method centered on the ramps in the merge bottleneck area, usually "single - point or local bottleneck" - oriented control and execution. In this patent, first, the upper layer outputs the section - level "total ramp quantity", and then the lower layer performs weighted allocation according to the congestion differences of the main - line sections corresponding to each ramp; this mechanism makes the control not limited to a single merge point, but rather faces section - level coordination of multiple bottleneck couplings.

[0031] Differences in the system implementation organization method;

[0032] Comparative Patent 3 includes "methods, systems, devices, and media" in the claim organization, emphasizing the full - set claim layout of the engineering implementation carrier. In this patent, the modular composition of the "hierarchical ramp coordination control system" (data acquisition, macroscopic state construction, upper - layer total - quantity decision - making, lower - layer allocation, ramp signal control, etc.) supporting the method has been given in the text, and its technical core still focuses on the collaborative control mechanism of "multi - bottleneck MFD macroscopic characterization + DRL upper - layer total - quantity control + lower - layer congestion weight allocation". Summary of the Invention

[0033] The present invention aims to overcome the deficiencies of the prior art. Aiming at the problems of strong dependence on model accuracy, insufficient global coordination, and high computational complexity in the multi - bottleneck ramp coordination control scenario of expressways, it provides a multi - bottleneck hierarchical ramp coordination control method for expressways based on deep reinforcement learning. Utilizing the macroscopic fundamental diagram theory, it realizes efficient global collaborative optimization of long - distance multi - bottleneck sections through a hierarchical ramp control architecture, and introduces a deep reinforcement learning method to improve the adaptability of the control method to the randomness of traffic operation.

[0034] To achieve the above - mentioned purpose, the technical solution adopted by the present invention is as follows:

[0035] A multi - bottleneck hierarchical ramp coordination control method for expressways based on deep reinforcement learning, including the following steps:

[0036] Step S1, at each discrete time step Based on the detection data, the units of each road segment are obtained. density ,flow And calculate the overall weighted average density of the main line. Weighted average flow and the standard deviation of density distribution ):

[0037] Overall weighted average density of the main line:

[0038]

[0039] Overall weighted average flow rate of the main line:

[0040]

[0041] Density distribution standard deviation:

[0042]

[0043] Where I represents the total number of units in the main line segment. For road segment units Length, Let i be the number of lanes in road segment unit i;

[0044] Step S2: Construction of a hierarchical ramp coordination control framework: The upper-level control aims to improve the overall traffic efficiency of the system and outputs control cycles. Expected total ramp flow The lower level controls the total flow rate of the ramps to not exceed [a certain value]. Under the constraints, with the goal of further improving local traffic efficiency, the flow rate of each entrance ramp is calculated;

[0045] Step S3, Training of the upper-level control ramp total flow control strategy: Model the upper-level control ramp total flow control problem as a Markov decision process, and construct a deep reinforcement learning framework with the overall index of the main line based on the macro basic graph and the total index of the ramps as the state, the expected total flow of the ramps entering the main line as the action, and the total outflow of the system as the reward.

[0046] For example, when the state and action space dimensions are low, a competitive dual deep Q-network (PER-D3QN) algorithm based on priority experience replay can be used. This algorithm is trained offline through repeated interaction with a multi-bottleneck simulation environment on a fast road to obtain an optimized control strategy for the total flow of the upper-level control ramps; the immediate reward is the total outbound flow of the system. While maintaining the definitions of state, action, and reward, other deep reinforcement learning algorithms such as TD3 can be used to replace the PER-D3QN algorithm depending on the complexity of the scenario.

[0047] Step S4, Online decision-making of total ramp flow in upper control: In actual or simulated scenarios, at the beginning of each control cycle, the state of the traffic system is calculated based on the data from the loop detectors. This state is then input into the trained deep reinforcement learning agent, which outputs the total ramp flow expected to enter the main line in the current control cycle.

[0048] As a further improvement to the coordinated control method of the present invention, the expressway includes urban expressways and highways, contains at least two densely distributed traffic bottlenecks, and is equipped with a series of detectors along the route, which are divided into sections based on the midpoint of adjacent detectors. Each road segment is divided into units to ensure close traffic connections within the segment.

[0049] As a further improvement to the coordinated control method of the present invention, the length of the multiple bottlenecks on the expressway is 10 to 20 km, and the recommended distance between adjacent bottlenecks is 0.8 to 4.8 km and does not exceed 8.0 km.

[0050] As a further improvement to the coordinated control method of the present invention, in step S3;

[0051] The state space of the upper-level control deep reinforcement learning agent is defined as follows:

[0052] in: The overall weighted average density of the main line. The standard deviation of the overall density distribution along the main line. The average length of the main line. The traffic demand at the starting point of the main line, The total number of vehicles accumulated on the ramp. For the total traffic demand of the ramps, This represents the total traffic flow of the ramps in the previous cycle.

[0053] As a further improvement to the coordinated control method of the present invention, in step S3:

[0054] (1) Upper-level control of deep reinforcement learning actions For: the total flow rate of the ramps expected to enter the main line during the current control cycle. The total flow rate of the ramp is discretely measured within a pre-set finite interval at fixed step sizes, forming a one-dimensional discrete action space.

[0055] Therefore, it is more appropriate to choose the discrete reinforcement learning method, which has higher computational efficiency and more stable convergence.

[0056] (2) Upper-level control deep reinforcement learning in each control cycle Instant rewards The total outbound traffic of the system includes outbound traffic from the mainline terminus and exit ramps. When the overall weighted average density of the mainline exceeds the key density threshold determined by the macroscopic basic chart, a preset penalty constant is subtracted from the original reward to accelerate the convergence speed of the agent. This invention provides a multi-bottleneck hierarchical ramp coordination control system for expressways based on deep reinforcement learning, used to implement a multi-bottleneck hierarchical ramp coordination control method for expressways based on deep reinforcement learning, comprising:

[0057] The data acquisition module is used to collect traffic data such as flow rate and density of each road segment, queue length of each entrance ramp, and ramp demand based on the loop detectors set in the multi-bottleneck control range of the expressway.

[0058] The macro-state construction module is used to calculate the overall weighted average density, weighted average flow, and density distribution standard deviation of the main line based on the traffic data, and to construct the overall macro-state of the main line. At the same time, it integrates the demand at the starting point of the main line, the cumulative total number of vehicles on the ramps, the total demand on the ramps, and the total flow of the ramps in the previous cycle to form the state input for the upper-level control reinforcement learning.

[0059] The upper-level control total flow decision module is used to deploy a ramp total flow control agent trained based on deep reinforcement learning. It receives the current cycle macro state and outputs the expected ramp total flow of the main line in the current cycle.

[0060] The lower-level control flow distribution module is used to calculate the total flow that needs to be restricted based on the demand of each ramp and the congestion index of the corresponding main line section, and to distribute the total flow of the ramps among each ramp according to the preset weight coefficient to obtain the actual flow of each ramp.

[0061] The ramp signal control module is used to convert the actual traffic flow of each ramp into the corresponding green light time of the ramp signal, and to send control commands to the ramp signal lights to be executed in the next control cycle.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) Benefiting from the theoretical basis of the macro-basic map: the overall macro state of multiple bottleneck sections is used as the basis for decision-making, and the global optimization of multiple bottlenecks is realized; it has the advantages of high robustness and low computational cost, and can meet the needs of large-scale road control.

[0064] (2) Adopting a model-free deep reinforcement learning method: reducing the dependence on accurate traffic prediction models, while determining that traffic prediction models and random actual traffic operations have an inherent mismatch; it can still maintain good control performance and robustness in scenarios of random fluctuations in demand, and is suitable for uncertain traffic demand conditions in actual operation. Attached Figure Description

[0065] Figure 1 This is a diagram of the multi-bottleneck layered ramp coordinated control architecture for expressways proposed in this invention;

[0066] Figure 2 The flowchart of ramp coordination control based on deep reinforcement learning (taking PER-D3QN as an example);

[0067] Figure 3 A schematic diagram of the simulated road network for a typical expressway with multiple bottlenecks.

[0068] Figure 4 A general macroscopic basic diagram of the main lines of different control schemes;

[0069] Figure 5 A graph showing the relationship between the mainline weighted average density and total ramp flow under different control schemes;

[0070] Figure 6 A comparison chart of total travel time for different control schemes under various demand fluctuation scenarios. Detailed Implementation

[0071] The following description, in conjunction with the accompanying drawings and embodiments, further illustrates the multi-bottleneck hierarchical ramp coordinated control method for expressways based on deep reinforcement learning of the present invention. It should be understood that the following embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of the present invention; those skilled in the art can make equivalent substitutions or adjustments to specific parameters and structures without departing from the concept of the present invention.

[0072] 1. Construction of Control Region and Macro-state

[0073] In a typical embodiment, a main expressway is selected as the control object, and the control range is the main expressway with a one-way length of approximately 15.1 km and its four pairs of entrance and exit ramps. The main expressway has four lanes, and each entrance ramp is of sufficient length with no risk of vehicle queuing overflow. Each entrance ramp is equipped with traffic lights to implement ramp control.

[0074] Multiple sets of detectors (such as coils, radar, or video detectors) are installed along the main line within the control area, and the main line is divided into several road segment units by the midpoint of adjacent detectors. During each control cycle, the detectors collect parameters such as traffic flow, vehicle speed, or occupancy rate for each road segment unit, and calculate the road segment density accordingly. Based on the density, flow rate, and length of each road segment unit, overall macroscopic status indicators of the main line are constructed, including the overall weighted average density, the overall weighted average flow rate, and the standard deviation of the overall density distribution. The specific calculation relationships for each indicator are as follows:

[0075] Overall weighted average density of the main line:

[0076]

[0077] Overall weighted average flow rate of the main line:

[0078]

[0079] Density distribution standard deviation:

[0080]

[0081] in, I is the number of discrete time steps, and I is the number of road segment units in the main line. Let i be the number of lanes in road segment unit i.

[0082] Simultaneously, the system can obtain the oncoming vehicle demand at the mainline starting point in the current control cycle, the number of vehicles queuing at each entrance ramp at the end of the previous cycle, and the traffic demand entering the ramps based on the detectors and ramp queuing detection devices. The aforementioned macroscopic state information and ramp traffic information together constitute the state input basis for the upper-level control deep reinforcement learning.

[0083] 2. Layered ramp coordination control framework

[0084] Based on the aforementioned control area and state construction, this invention constructs a hierarchical ramp coordination control framework. Its overall system architecture is as follows: Figure 1 As shown.

[0085] The upper-level control layer is the global ramp total flow decision layer. Taking the overall macroscopic state of the mainline and ramp traffic information as input, it outputs the expected ramp total flow entering the mainline in each control cycle. The objective of the upper-level control is to maximize the outbound flow of the traffic system, thereby reducing the total travel time of the system.

[0086] The lower-level control is a local total flow allocation layer. Under the total flow constraint of the ramps given by the upper-level control, the lower-level control allocates flow to each ramp according to the actual demand of each entrance ramp and the congestion level of its corresponding mainline section. Ramps with high congestion levels use lower flow to alleviate congestion in the corresponding bottleneck area; ramps with low congestion levels use higher flow to make full use of the mainline capacity.

[0087] 3. Design and Training of the Upper-Level Control Ramp Total Flow Decision Agent

[0088] The decision-making process for total flow of upper-level control ramps based on the macroscopic basic diagram is as follows: Figure 2 As shown in the figure. Among them, the deep reinforcement learning algorithm PER-D3QN is used as an example. In specific control applications, other algorithms can be selected according to the complexity of the scenario.

[0089] (1) Reinforcement learning modeling and state, action, and reward design

[0090] In this embodiment, the overall traffic flow control problem of the upper-level control ramps is modeled as a discrete-time Markov decision process. In each control cycle, the state vector is composed of seven indicators in a fixed order: the overall weighted average density of the mainline, the standard deviation of the overall density distribution of the mainline, the overall average travel length of the mainline, the demand at the starting point of the mainline, the total number of vehicles queuing at all entrance ramps, the total demand at all entrance ramps, and the actual total traffic flow of the ramps in the previous control cycle.

[0091]

[0092] in: The overall weighted average density of the main line. The standard deviation of the overall density distribution along the main line. The average length of the main line. The traffic demand at the starting point of the main line, The total number of vehicles accumulated on the ramp. For the total traffic demand of the ramps, This represents the total traffic flow of the ramps in the previous cycle.

[0093] Action is defined as the total flow of ramps expected to enter the main line during the current control cycle, i.e. There are two ways to obtain the action: one is to directly use the obtained integer value as... Secondly, the obtained values ​​are amplified and used as... To meet the requirements of flow dispersion in actual control, the total flow is discretized within a pre-defined interval with a fixed step size, forming a one-dimensional discrete action space.

[0094] The reward settings aim to minimize the total system travel time, assuming traffic demand is independent of the control scheme. This embodiment assigns rewards to the agent for each control cycle. Rewards Defined as total outbound flow (veh / s), including outbound flow from the mainline terminus and exit ramps. Meanwhile, to accelerate convergence, based on the MFD principle for expressways, when the overall weighted average density of the mainline significantly exceeds its critical value (e.g., exceeding 20 veh / km), a fixed positive number (e.g., 3) is subtracted from the original reward.

[0095] (2) Offline training and online application process

[0096] This invention constructs a multi-bottleneck road network corresponding to actual expressways on a microscopic simulation platform. Based on measured traffic data, it generates multiple sets of weekday evening peak demand sequences and several random disturbance scenarios for training and testing. The training process includes: in each control cycle, calculating a state vector based on detector data output from the simulation environment and inputting it into the agent; the agent then... - A greedy strategy selects the total traffic flow action for each ramp, which is then allocated to and executed by the lower-level control module. After the control cycle ends, the environment provides new status and immediate rewards, forming experience samples stored in the experience pool. When the number of samples in the experience pool reaches a threshold, a small batch of experiences is sampled according to a priority experience replay mechanism to calculate the target. The values ​​and weighted loss function are used to update the online network parameters, and the target network is periodically updated according to the soft update rules.

[0097] As the number of training rounds increases, the total reward generally shows an upward trend, while the total system travel time gradually decreases and converges to a stable level. After offline training is completed, the trained PER-D3QN network is deployed in the upper-layer control total flow decision module. In the actual operation or online simulation phase, the current traffic state is read at a fixed control cycle, and the ramp total flow control quantity for the current cycle can be obtained in real time through forward calculation. The computational load is relatively small and can meet the requirements of real-time applications.

[0098] 4. Lower-level control flow distribution and ramp signal control

[0099] (1) Calculation of the total flow of the ramps that need to be restricted

[0100] After the upper-level control agent outputs the total ramp flow for the current control cycle, the lower-level control module first calculates the total flow that needs to be restricted for all entrance ramps in that cycle. The difference between the sum of the demand flow for all controlled entrance ramps and the expected total ramp flow for the upper-level control is the total flow that needs to be restricted; when the total demand does not exceed the expected value for the upper-level control, the total flow that needs to be restricted is zero. The specific calculation of the total flow that needs to be restricted is as follows:

[0101]

[0102] in, To control the period, it consists of several discrete time steps. composition, It's a ramp The flow rate unaffected by ramp control can be estimated using the following formula:

[0103]

[0104] in, ramp Traffic capacity, It's a ramp The number of vehicles in the queue It's a ramp The traffic capacity of the upstream main line. It's a ramp The maximum received flow rate in the downstream confluence zone can be estimated using the cellular transport model (CTM) based on the current density of that zone.

[0105] (2) Flow allocation weights based on congestion indicators

[0106] To reflect the differences in congestion levels across mainline sections corresponding to different ramps, the lower-level control module assigns a weighting coefficient to each entrance ramp. Since there is no need to restrict the flow of traffic at corresponding entrance ramps in uncongested mainline sections (assuming no queuing overflow risk at each ramp), the corresponding weight is 0. For congested mainline sections, the weighting coefficient increases as the congestion level of the corresponding section increases. Based on the macroscopic basic graph theory, the congestion level of each mainline section is comprehensively characterized by the weighted average density and density distribution difference index of that section. The specific calculation relationship of the weighting coefficient is as follows:

[0107]

[0108] in, ramp The set of road segment units corresponding to the main line section. and These are the maximum and minimum density values ​​of the road segment units within the main line section, respectively. ramp The weighted average density corresponding to the main line segment, This is an indicator of density distribution differences within the main line section.

[0109] The total traffic flow needs to be limited and distributed among the ramps according to a weighted coefficient. The actual traffic flow of each ramp in the next control cycle... The uncontrolled demand traffic is deducted from the weighted, throttling traffic, and the relevant calculation relationship is as follows:

[0110]

[0111] Through the above allocation process, stricter flow restrictions can be implemented on the more congested ramps on the downstream main line, thereby alleviating local bottleneck congestion and improving the overall traffic efficiency of the main line.

[0112] (3) Execution of ramp signal control

[0113] The calculated actual traffic flow for each ramp is further converted into the green light time for the ramp signal. This is done after obtaining the green light time for each individual entrance ramp. Traffic Then, convert it into a green light time. The ramp signal control module generates control commands based on the conversion results, sends them to the ramp signal lights on site for execution, and executes them in the next control cycle.

[0114]

[0115] in, ramp The set of road segment units corresponding to the main line section.

[0116] 5. Simulation Examples and Effect Comparison

[0117] To verify the effectiveness and superiority of the method of this invention, a comparative experiment was conducted using a microscopic simulation platform under the aforementioned typical multi-bottleneck expressway network. The simulation scenario settings included: selecting approximately 2 hours of evening rush hour on weekdays as the main simulation period, with a warm-up time of approximately 20 minutes; updating traffic demand in 15-minute increments; generating multiple sets of demand sequences based on a Gaussian distribution random perturbation using a given average demand curve to simulate the random fluctuations in actual traffic demand; and generating several different random perturbation sequences for each average demand level, with repeated simulations corresponding to multiple "weekdays." The typical multi-bottleneck expressway network used in this embodiment is as follows: Figure 3 As shown.

[0118] The control schemes compared include: no control scheme (NC), model predictive control scheme based on macroscopic basic map (MFD-MPC), and the deep reinforcement learning hierarchical ramp coordination control scheme (MFD-DRL) proposed in this invention, where the upper-level control is based on PER-D3QN for ramp total flow decision. The lower-level control of both the MFD-MPC and MFD-DRL schemes is the aforementioned flow allocation method based on congestion weights. Under preset traffic demand, the overall macroscopic basic map of each scheme is compared as follows: Figure 4 As shown; the relationship between the total flow rate of ramps with different control schemes and the weighted average density is as follows: Figure 5 As shown, this illustrates the relationship between control actions and system performance during the control process. The overall evaluation index is the total travel time (TTS) of the system. The TTS comparisons of different control schemes under different traffic demand fluctuation levels are illustrated below. Figure 6 As shown.

[0119] Simulation results show that, under the preset traffic demand, the overall operation of the NC scheme's main line exhibits a typical evolution of "density accumulation leading to oversaturation—congestion dissipation forming a hysteresis loop." Figure 4 (a) The MPC scheme basically avoids the main line from falling into oversaturation before 18:20, and the overall situation is relatively stable. Figure 4 (b) The total flow rate of its ramps is dynamically adjusted according to the weighted average density of the main line to suppress excessively rapid density increases. Figure 5 (a) However, during the congestion recovery phase, especially when the weighted average density is low (e.g., about 80 veh / km, less than the critical density of 90 veh / km) and there are discrepancies between the prediction model and the actual situation, local congestion may still be induced, leading to fluctuations in the control effect. Figure 5(b) The DRL scheme can implement appropriate total flow control at the right time to maintain the weighted average density of the mainline near the critical density during peak hours, while taking into account both mainline operation and ramp queuing; for example, around 17:30, when the weighted average density is slightly higher than the critical density, DRL can significantly reduce the overall ramp queuing while allowing a small loss of weighted average flow. Its "density-total flow" adjustment trajectory is shown in [reference needed]. Figure 5 (b) The overall evolution of the main MFD is shown in [reference]. Figure 4 (c).

[0120] Under different traffic demand fluctuation scenarios ( Figure 6 Demand sequences were generated using a Gaussian distribution, and increasing fluctuation intensities were formed by selecting standard deviations of 0%, 5%, 10%, 15%, and 20% of the mean. Ten operating scenarios were generated for each intensity, and the average TTS was compared. The results showed that the TTS of the NC scheme increased significantly with increasing fluctuation; the MPC scheme performed better in low-fluctuation scenarios, but its performance gradually deteriorated with increasing fluctuation; the DRL scheme, based on model-free deep reinforcement learning to obtain ramp total volume control strategies, was more adaptable to traffic operation uncertainty and random fluctuations in demand, and maintained the best (or better) TTS performance in all the above fluctuation scenarios.

[0121] In summary, the multi-bottleneck hierarchical ramp coordinated control method for expressways proposed in this invention, without the need to explicitly construct an accurate traffic prediction model, can effectively improve the overall traffic efficiency of long-distance multi-bottleneck expressway sections and maintain good control robustness under random fluctuations in traffic demand, thus possessing high engineering application value.

[0122] 6. System Implementation Description

[0123] The method of this invention can be implemented through a combination of hardware and software. Typically, a hierarchical ramp coordination control system can be deployed in the coil or control center, including: a data acquisition module for collecting mainline and ramp detection data; a macroscopic state construction module for calculating macroscopic indicators and constructing upper-level control state inputs; an upper-level control total flow decision module for running the PER-D3QN agent and outputting the total ramp flow; a lower-level control flow allocation module for allocating flow under total constraints and generating the flow of each ramp; and a ramp signal control module for converting flow into ramp signal control commands. These modules can be implemented using industrial computers, embedded controllers, or servers, and interact with the on-site ramp traffic lights via a communication network for data exchange and command issuance.

[0124] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multi-bottleneck hierarchical ramp coordinated control method for fast roads based on deep reinforcement learning, characterized in that, Includes the following steps: Step S1: At each discrete time step Based on the detection data, the units of each road segment are obtained. density ,flow And calculate the overall weighted average density of the main line. Weighted average flow and the standard deviation of density distribution ): Overall weighted average density of the main line: ； Overall weighted average flow rate of the main line: ； Density distribution standard deviation: ； Where I is the total number of road segment units in the main line. For road segment units Length, Let i be the number of lanes in road segment unit i; Step S2: Construction of a hierarchical ramp coordination control framework: The upper-level control aims to improve the overall traffic efficiency of the system and outputs control cycles. Expected total ramp flow The lower level controls the total flow rate of the ramps to not exceed [a certain value]. Under the constraints, with the goal of further improving local traffic efficiency, the flow rate of each entrance ramp is calculated; Step S3, Training of the upper-level control ramp total flow control strategy: Model the upper-level control ramp total flow control problem as a Markov decision process, and construct a deep reinforcement learning framework with the overall macro-based basic graph index of the main line and the total ramp index as the state, the expected ramp total flow of the main line as the action, and the total system outflow as the reward. Step S4, Online decision-making of total ramp flow in upper control: In actual or simulated scenarios, at the beginning of each control cycle, the state of the traffic system is calculated based on the data from the loop detectors. This state is then input into the trained deep reinforcement learning agent, which outputs the total ramp flow expected to enter the main line in the current control cycle.

2. The method for coordinated control of multi-bottleneck hierarchical ramps on expressways based on deep reinforcement learning according to claim 1, characterized in that, The expressway includes urban expressways and highways, contains at least two densely distributed traffic bottlenecks, and is equipped with a series of detectors along its route, divided into sections based on the midpoint of adjacent detectors. Each road segment unit.

3. The method for coordinated control of multi-bottleneck hierarchical ramps on expressways based on deep reinforcement learning according to claim 1 or 2, characterized in that, The length of the bottlenecks on the expressway is 10 to 20 km, and the recommended distance between adjacent bottlenecks is 0.8 to 4.8 km and does not exceed 8.0 km.

4. The method for coordinated control of multi-bottleneck hierarchical ramps on expressways based on deep reinforcement learning according to claim 1 or 2, characterized in that, In step S3; The state space of the upper-level control deep reinforcement learning agent is defined as follows: ; in: The overall weighted average density of the main line. The standard deviation of the overall density distribution along the main line. The average length of the main line. The traffic demand at the starting point of the main line, The total number of vehicles accumulated on the ramp. For the total traffic demand of the ramps, This represents the total traffic flow of the ramps in the previous cycle.

5. The method for coordinated control of multi-bottleneck hierarchical ramps on expressways based on deep reinforcement learning according to claim 1 or 2, characterized in that, In step S3: (1) Upper-level control of deep reinforcement learning actions For: the total flow rate of the ramps expected to enter the main line during the current control cycle. The total flow rate of the ramp is discretely measured within a pre-set finite interval at fixed step sizes, forming a one-dimensional discrete action space. (2) Upper-level control deep reinforcement learning in each control cycle Instant rewards The total outbound traffic of the system includes outbound traffic from the end of the main line and the exit ramp. When the overall weighted average density of the main line exceeds the key density threshold determined by the macroscopic basic chart, a preset penalty constant is subtracted from the original reward to accelerate the convergence speed of the agent.

6. A multi-bottleneck hierarchical ramp coordination control system for fast roads based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 5, characterized in that, include: The data acquisition module is used to collect traffic data such as flow rate and density of each road segment, queue length of each entrance ramp, and ramp demand based on the loop detectors set in the multi-bottleneck control range of the expressway. The macro-state construction module is used to calculate the overall weighted average density, weighted average flow, and density distribution standard deviation of the main line based on the traffic data, and to construct the overall macro-state of the main line. At the same time, it integrates the demand at the starting point of the main line, the cumulative total number of vehicles on the ramps, the total demand on the ramps, and the total flow of the ramps in the previous cycle to form the state input for the upper-level control reinforcement learning. The upper-level control total flow decision module is used to deploy a ramp total flow control agent trained based on deep reinforcement learning. It receives the current cycle macro state and outputs the expected ramp total flow of the main line in the current cycle. The lower-level control flow distribution module is used to calculate the total flow that needs to be restricted based on the demand of each ramp and the congestion index of the corresponding main line section, and to distribute the total flow of the ramps among each ramp according to the preset weight coefficient to obtain the actual flow of each ramp. The ramp signal control module is used to convert the actual traffic flow of each ramp into the corresponding green light time of the ramp signal, and to send control commands to the ramp signal lights to be executed in the next control cycle.

Images