Traffic signal control method based on regional hierarchical multi-agent reinforcement learning

Abstract

The invention discloses a traffic signal control method based on regional hierarchical multi-agent reinforcement learning, and belongs to the technical field of intelligent traffic systems. According to the method, a target traffic network is divided into a plurality of areas, each area is configured with a management layer agent, each intersection in the area is configured with a control layer agent, and a double-layer layered architecture is constructed; the management layer intelligent agent periodically generates a regional control target based on regional joint state observation, and the target remains unchanged in a plurality of bottom layer control periods; the control layer intelligent agent collects local state observation in each control period, receives the regional control target, dynamically corrects the target based on a local congestion state, and outputs a signal lamp phase control action; the control layer and the management layer respectively calculate level rewards, empirical data are stored in an empirical playback buffer area, and an offline strategy training mode is combined with a double-Q network, Huber loss, delay update and an action entropy regularization mechanism to collaboratively optimize two layers of strategy functions.

Description

Technical Field

[0001] This invention relates to the field of traffic control technology, and more specifically to a traffic signal control method based on regional hierarchical multi-agent reinforcement learning. Background Technology

[0002] With the acceleration of global urbanization, traffic congestion and environmental pollution are becoming increasingly serious problems. Given the constraints of limited investment in road infrastructure, improving urban traffic control (UTC) efficiency through intelligent optimization of traffic signal timing has become a core development direction for Intelligent Transportation Systems (ITS). Traditional traffic signal control methods mainly include two categories: fixed timing and inductive control. The former relies on historical traffic data to preset cycles, resulting in poor adaptability in dynamic traffic environments; the latter, such as SCATS and SCOOTS, while able to dynamically adjust based on detector data, is still limited by expert rules and struggles to cope with complex scenarios such as sudden congestion and multi-source heterogeneous traffic flows.

[0003] In recent years, deep reinforcement learning (DRL) has demonstrated significant advantages in traffic signal control (TSC) due to its powerful environmental perception and adaptive decision-making capabilities. In particular, multi-agent reinforcement learning (MARL) achieves distributed cooperative control by deploying independent agents at each intersection, effectively mitigating the "curse of dimensionality" problem of single-agent methods in large-scale road networks. However, existing MARL schemes still face three key challenges: first, the coarse design of states and rewards leads to sparse policy learning feedback and slow convergence; second, as the road network expands, the joint state-action space grows exponentially, making it difficult to balance model scalability and control performance; and third, the synchronous updating of multi-agent policies causes environmental non-stationarity, severely hindering stable convergence of the algorithm.

[0004] To overcome these bottlenecks, researchers have attempted to introduce a hierarchical reinforcement learning (HRL) framework to decompose complex tasks through spatiotemporal abstraction. However, most existing HRL methods either focus only on task decomposition in the temporal dimension or employ coarse-grained region partitioning, failing to effectively establish a dynamic collaborative mechanism between high-level goal guidance and low-level action execution. Especially in time-sensitive scenarios such as priority passage for emergency vehicles, existing methods lack the ability to respond quickly to special traffic demands, often sacrificing overall road network efficiency for local priority, making it difficult to achieve a balance between multiple objectives.

[0005] Therefore, how to provide a multi-agent reinforcement learning model based on a small-area hierarchical architecture, and construct a two-layer collaborative system of "management layer-control layer" through spatial road network partitioning and temporal target-correction mechanism, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, this invention is proposed to provide a traffic signal control method based on regional hierarchical multi-agent reinforcement learning to overcome or at least partially solve the above problems. This invention, through regional hierarchical architecture and target dynamic correction mechanism, significantly improves the performance and scalability of signal control in large-scale road networks, while efficiently ensuring priority passage for emergency vehicles without significantly affecting social traffic flow, and achieves the organic unity of global coordination, local adaptation and multi-objective optimization.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Firstly, a traffic signal control method based on regional hierarchical multi-agent reinforcement learning includes: The target traffic network is divided into regions, and a management layer agent is configured for each region, while a control layer agent is configured for each intersection within the region. During each control cycle, each control layer agent collects the lane density of its corresponding intersection approach lane and receives the area control target issued by its management layer agent. Each control layer agent dynamically corrects the received area control target based on the lane density of the intersection approach lanes to obtain the corrected control target, and outputs a traffic light phase control action through the control layer strategy function based on the corrected control target, and executes the action. Each control layer agent calculates the control layer reward based on its control effect and stores the experience data in the experience replay buffer. The management layer agent aggregates the state observations of all control layer agents within its jurisdiction to form a regional joint state observation. The management layer agent, based on the joint state observation of the region, outputs regional control objectives through the management layer policy function to guide the control layer, and periodically calculates management layer rewards based on the overall traffic operation effect of the region. Using the data in the experience playback buffer, the control layer policy function and the management layer policy function are updated respectively using an offline policy training method to optimize the overall traffic signal control performance.

[0009] Preferably, the step of dividing the target traffic network into regions specifically includes: Abstract the target traffic network into a graph structure, where nodes represent intersections and edges represent roads; Obtain the geographic coordinates of each intersection; The K-Means clustering algorithm is used to divide all intersections into several regions based on their geographical coordinates.

[0010] Preferably, the dynamic correction calculation formula is as follows:

[0011] in, The corrected control target, They are respectively and Lane density of the approach lanes at the intersection at any given time. Indicates the area control objective.

[0012] Preferably, the control layer reward includes rewards for social vehicles and rewards for emergency vehicles, calculated using the following formula:

[0013] in, For control layer intelligent agents Total reward This indicates a reward for private vehicles. These are the weighting coefficients. Rewards for emergency vehicles, for Enter the control layer intelligent agent at any time The number of emergency vehicles within the controlled area;

[0014]

[0015] in, Speed ​​limits for lanes, Minimum driving speed, The average speed of social vehicles. The average speed of the emergency vehicle. For the number of lanes, and These represent the queue lengths at the previous and next time points, respectively. and These represent the lane vehicle density at the preceding and following times, respectively. , These are the weighting coefficients.

[0016] Preferably, the empirical data are as follows:

[0017] in, Empirical data collected to represent the control agent's policy. For control layer intelligent agents Total reward This represents the regional control target at time t. This represents the regional control target at time t+1. This represents the state observation at time t. This represents the state observation at time t+1.

[0018] Preferably, the state observation of the control layer agent includes at least one or more of the following: The intersection edge density, the queue length of vehicles in each approach lane of the intersection, the maximum / minimum green light phase, the current traffic light phase, the lane density of the approach lanes of the intersection, the approach lane density difference, the number of emergency vehicles in the approach lanes, the weighted time-varying average speed of emergency vehicles in the approach lanes, the distance of emergency vehicles to the intersection, and the waiting time of emergency vehicles.

[0019] Preferably, the current management agent balances the rewards of other management agents to fully balance the inter-regional collaboration mechanism, and the management layer reward is:

[0020]

[0021] in, To coordinate the reward coefficients of the current management agent and neighboring agents, Indicates the region A collection of control agents, Represents the control layer intelligent agent Total reward It is a region The set of adjacent regions.

[0022] Preferably, the control layer agent Critic target network:

[0023] in, The Critic target network represents the controlling agent. As a discount factor, Represents the control layer intelligent agent The total reward at time t, and Indicates the target Q value. This represents the state observation at time t+1. This represents the area control target at time t+1. Indicates the operation of the traffic light phase control; The management agent Critic target network is defined as follows:

[0024] in, The Critic target network represents the management agent. This represents the reward for the management agent at time t. and Indicates the target Q value. This represents the joint state observation of the region at time t+1.

[0025] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a traffic signal control method based on regional hierarchical multi-agent reinforcement learning, which has the following effects: 1. Significantly improves the scalability and control performance of large-scale road networks. By dividing the traffic network into several small areas and deploying a management layer agent in each area to coordinate multiple control layer agents, the "curse of dimensionality" and non-stationarity problems faced by traditional flat multi-agent methods in large-scale road networks are effectively alleviated. The hierarchical structure reduces the coupling complexity between agents, enabling the system to scale to road networks of any size as needed, while maintaining stable training convergence and real-time control capabilities.

[0026] 2. Achieve organic synergy between high-level strategic guidance and low-level adaptive execution. The management outputs regional control targets (such as traffic priority direction and congestion mitigation intention) on a slower time scale, and the control layer dynamically corrects the targets in each control cycle based on local conditions (especially lane density). This mechanism enables the system to both implement global optimization intentions (such as regional green wave coordination) and flexibly respond to localized sudden congestion or traffic fluctuations. The dynamic correction mechanism enhances the robustness of the strategy and maintains good performance under different traffic densities (light load, saturation, and oversaturation).

[0027] 3. Efficiently support the priority passage of high-priority traffic flows such as emergency vehicles. By incorporating emergency vehicle status information into the control layer state observations and setting an asymmetric reward term positively correlated with emergency vehicle speed in the reward function, the hierarchical strategy automatically learns during training that when an emergency vehicle appears, the management layer tends to generate regional targets that promote its passage, while the control layer adaptively adjusts these targets based on real-time congestion conditions to achieve efficient priority passage. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0029] Figure 1 This is a flowchart of a traffic signal control method based on regional hierarchical multi-agent reinforcement learning provided in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of a traffic signal control method based on regional hierarchical multi-agent reinforcement learning provided in an embodiment of the present invention.

[0031] Figure 3 This is a structural diagram of the dual-layer HRL road network traffic light control model provided in this embodiment of the invention.

[0032] Figure 4 This is a schematic diagram of a two-way, two-lane adjacent intersection model provided in an embodiment of the present invention.

[0033] Figure 5 This is a maximum / minimum green light phase rule diagram provided in an embodiment of the present invention.

[0034] Figure 6 This is a schematic diagram of one-hot encoding of phase action / state provided in an embodiment of the present invention.

[0035] Figure 7 This is a heat map showing the maximum speed limit of a 6x6 road network provided in this embodiment of the invention.

[0036] Figure 8 This is a 6x6 road network traffic flow curve provided in an embodiment of the present invention.

[0037] Figure 9 This refers to the cumulative average reward of the 6x6 road network control layer provided in this embodiment of the invention.

[0038] Figure 10 This refers to the cumulative average reward for the 6x6 road network management layer provided in this embodiment of the invention.

[0039] Figure 11 The average vehicle speed of the 6x6 road network provided in this embodiment of the invention.

[0040] Figure 12 This refers to the cumulative waiting time for the 6x6 road network provided in this embodiment of the invention.

[0041] Figure 13 This refers to the average waiting time for the 6x6 road network provided in this embodiment of the invention.

[0042] Figure 14 This refers to the cumulative number of parking spaces on the 6x6 road network provided in this embodiment of the invention.

[0043] Figure 15 The travel time for the 6x6 road network provided in this embodiment of the invention.

[0044] Figure 16 This refers to the 6x6 road network integrated training index provided in this embodiment of the invention.

[0045] Figure 17The average speed of emergency vehicles on the 6x6 road network provided in this embodiment of the invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention discloses a traffic signal control method based on regional hierarchical multi-agent reinforcement learning, such as... Figure 1 and Figure 2 As shown, it includes: The target traffic network is divided into regions, and a management layer agent is configured for each region, while a control layer agent is configured for each intersection within the region. During each control cycle, each control layer agent collects the lane density of its corresponding intersection approach lane and receives the area control target issued by its management layer agent. Each control layer agent dynamically corrects the received regional control target based on the lane density of the intersection approach lanes to obtain the corrected control target, and outputs the traffic light phase control action through the control layer policy function based on the corrected control target, and executes the action. Each control layer agent calculates the control layer reward based on its control effect and stores the experience data in the experience replay buffer. The management layer agent aggregates the state observations of all control layer agents within its jurisdiction to form a joint regional state observation. The management agent, based on joint state observations of the region, outputs regional control objectives through the management policy function to guide the control layer, and periodically calculates management rewards based on the overall traffic operation performance of the region. By utilizing data from the experience replay buffer, the control layer policy function and the management layer policy function are updated separately using an offline policy training method to optimize the overall traffic signal control performance.

[0048] In this embodiment, the target traffic network is divided into regions, and a management layer agent is configured for each region. Simultaneously, a control layer agent is configured for each intersection within each region, including: Abstract the target transportation network into a single map. ,in Represents a set of nodes (a set of intersections). This represents the edge set (road or connecting edges). Building upon this, to reduce the collaborative complexity of Multi-Agent Reinforcement Learning (MARL), a regionalized control framework based on Hierarchical Reinforcement Learning (HRL) is proposed: obtaining the geographical coordinates of each intersection, and using the K-Means algorithm to process the node set... The system is divided into regions to ensure that nodes within the same region have high similarity. Clustering is based on each node. Position coordinates Each cluster region is assigned a management agent, while each intersection within the region is assigned a control agent.

[0049] The dual-layer HRL (Hierarchical Spatiotemporal Signal) road network signal control model designed in this invention adopts the FHAR (Feudal HRL) architecture. For example... Figure 3 As shown, the model is divided into control layers from bottom to top. and management At the micro level, each intersection in the control layer corresponds to an agent responsible for perceiving the local state, executing traffic light phase control tasks, and achieving collaborative optimization within a small area. At the macro level, the management layer agents operate on a regional basis, with each agent managing the control layer agents within its region. The agent's unit information is composed of the control agent elements of its small region, and it is responsible for formulating regional goals at a higher level of time and behavior abstraction, guiding the policy learning of the control layer agents.

[0050] In this embodiment, the control layer policy function is first... Observe the local state And receive management strategies Issued regional control targets ,exist Within steps of the target area It remains unchanged. Secondly, the control layer policy remains unchanged. Within one step, target correction is performed based on environmental feedback, and the correction transition function is:

[0051] in, This indicates the control target after correction. and Represent Time and The ideal state component at any given time is specifically defined as the lane density of the intersection approach lanes, used to measure the congestion level of the road network. Mathematically, and These are from the original observation vectors and The subset extracted from, i.e. and .also, The overall control objectives at the regional level were characterized. Based on the correction objectives... The control layer intelligent agent's strategy outputs signal light phase control actions. .

[0052] In this embodiment, each control layer agent calculates the control layer reward based on its control effect and stores the experience data in the experience replay buffer, specifically including: 1. The control effect of the control layer agent corresponds to the control effect of the traffic flow around a specific intersection. To provide real-time and accurate feedback on the control effect of the control layer agent, the reward obtained by the control layer agent consists of two parts: rewards for regular vehicles and rewards for emergency vehicles. Represented as:

[0053] For EMV, this invention introduces lane speed limits. Average speed of social vehicles and the average speed of emergency vehicles To construct the immediate relative velocity priority reward function for EMV :

[0054] In the above formula, to prioritize the speed of emergency vehicles, this invention uses the average speed of emergency vehicles. Average speed of social vehicles The difference method, but only considering the difference There is a risk of target misalignment, causing the system to overemphasize speed and neglect the importance of speed. For optimization, an asymmetric difference form was therefore adopted. .

[0055] In summary, the control layer intelligent agent The system reward function is:

[0056] in, For control layer intelligent agents The total reward at time t, This indicates a reward for private vehicles. These are the weighting coefficients. Rewards for emergency vehicles, for Enter the control layer intelligent agent at any time The number of emergency vehicles within the control range; in the absence of emergency vehicles, the system only includes a reward function for social vehicles, where each control agent corresponds to one intersection, and the number of both is the same.

[0057] 2. Calculate the reward for the control layer agent. Current state observation Control target Traffic light phase control action and the state observation at the next moment. and target The input strategy experience trajectory is represented as:

[0058] In the above formula, The empirical data collected by the control agent's policy is used, on the one hand, in the control layer policy function. On the one hand, there are updates to the data, and on the other hand, there is empirical data used to construct management agents.

[0059] In this embodiment, the management layer agent aggregates the state observations of all control layer agents within its jurisdiction to form a joint regional state observation; including: 1. The system constructs a regional joint state observation of the management layer agent by aggregating the state observations of the controlled agents under its jurisdiction. , represented as:

[0060] In the formula, This represents the vector concatenation operator, and the number of agents in the region control layer is... , and Representing the 1st and 2nd in the region respectively State observation of each control layer agent.

[0061] 2. The previous paragraph explained the joint state observation of the region. The construction process of the control agent within the region is illustrated using a two-way two-lane road as an example. Figure 4 Middle, intersection and between lanes These constitute common observed variables; for a specific intersection, the green arrow indicates... The red arrows indicate the inflow traffic attributes (density, queue length) at the corresponding intersection. This refers to the outbound traffic flow attributes at the corresponding intersection. (Based on the intersection...) For example, the set of edges it controls is represented as: For ease of description, the letters "intersection agent" are omitted. abbreviated as Number of lanes for entry and exit Among them, the corresponding set of lane sides. Gathering by the exit lane .

[0062] In this embodiment, the state observation selection for the control agent corresponding to a single intersection includes the following parts: 1) Edge density; side vehicle density for:

[0063] in, It represents the number of vehicles contained in the edge, and min_Gap is the minimum vehicle distance. It is the average length of the vehicle. This is the current lane length.

[0064] Therefore, the density vector of the edges at a certain intersection can be expressed as:

[0065] 2) Vehicle queue length;

[0066] in, This represents the maximum length of the parking queue in the edge.

[0067] 3) Maximum / minimum green light phase; The maximum / minimum phase indicator represents the interval between adjacent green light phase transitions, measuring whether the signal switching frequency is within a reasonable range. The calculation rules are as follows: Figure 5 As shown, it is expressed as the following formula:

[0068] in, and These represent the minimum and maximum green light phase indicators, respectively. For example, This indicates that the phase switching interval is within the expected and reasonable range; 4) Traffic light phase; Current traffic light phase (Excluding phases with only a transitional yellow light) is equivalent to the action of the previous moment. It is an important environmental status observation indicator, represented in the form of one-hot coding, such as Figure 6 As shown.

[0069] 5) Lane density of approach lanes at intersections; side Lane density of the approach lane at the intersection for:

[0070] in, Indicates the lane to enter The number of vehicles.

[0071] 6) Lane density difference;

[0072] in, and This represents the vehicle density at time t and time t-1.

[0073] 7) Number of emergency vehicles (considering only the approach lane); In reality, the number of emergency vehicles deployed varies depending on the level of disaster relief. Therefore, segmentation is necessary during the state mapping process. The number of vehicles in a given entry lane at an intersection is roughly divided into three segments, and the emergency vehicle quantity segmentation function is defined as follows: Represented as:

[0074] in, lane The actual number of emergency vehicles. Based on the above piecewise function, the normalized vector of the number of emergency vehicles at the intersection is:

[0075] 8) Weighted time-varying average speed of emergency vehicles (considering only the approach lane); When the number of emergency vehicles ,Lane The speed of emergency vehicles is calculated as follows:

[0076] in, Indicates the maximum speed. and Indicates the emergency vehicle at time t and time t-1. speed, The time interval is between adjacent periods. When the number of emergency vehicles... ,but .

[0077] In summary, the speed state vector of the emergency vehicle at the intersection is represented as:

[0078] 9) Distance between emergency vehicle and intersection center; The normalized distance from the emergency vehicle to the intersection is shown in the following formula, when a certain lane Number of emergency vehicles In order to emphasize the "urgency" of multiple emergency vehicles approaching, a linked-up approach is adopted. The distance is attenuated in a certain way; when The normalized distance is 1, representing a scenario without urgent passage requirements.

[0079]

[0080] in, This represents the distance from the first emergency vehicle in the line to the center of the intersection at time t.

[0081] 10) Emergency vehicle waiting time; Emergency vehicle waiting time is an important indicator of the operational status of emergency vehicles. Similar to the method of segmenting the number of emergency vehicles, the following piecewise function is used to map the waiting time to... between:

[0082]

[0083] in, This indicates that the emergency vehicle is in the lane at time t. The waiting time.

[0084] In this embodiment, the management agent, based on joint state observations of the region, outputs regional control objectives through a management policy function to guide the control layer, and periodically calculates management rewards based on the overall traffic operation performance of the region, including: 1. The management-level intelligent agent observes the region. Joint state observation of agent regions By policy function Mapped area The policy objective of each control layer agent ,in Represents the strategy target space.

[0085] 2. In this embodiment, the control task is to improve traffic operation efficiency, corresponding to the task level division of the management layer and the control layer. The task of the management layer reward in this invention is to guide macro-level coordination between regions and maximize the throughput of the entire road network. Accordingly, the designed management layer reward is the sum of the rewards of the control agents within the region, expressed as:

[0086] in, Indicates the region A collection of control agents, Represents the control layer intelligent agent The total reward. To balance rewards across regions, the total reward of other regions is considered, and the region... Adjusted rewards Designed as follows:

[0087] In the above formula, It is a region The set of adjacent regions, This is a coefficient that coordinates the rewards of the current management agent and neighboring agents. For the management agent f corresponding to region j, the corresponding reward for management agent f is... .

[0088] In this embodiment, the control layer policy function and the management layer policy function are updated respectively using the data in the experience playback buffer and the offline policy training method to optimize the overall traffic signal control performance.

[0089] This invention employs a dual-network architecture of actor-critic in both the management and control layers, with the AC algorithm policy updated based on approximate value estimates from the critics. However, the overestimation of the value function is a key factor leading to instability in policy updates. Therefore, this invention designs an independent dual-Q network to construct the target value:

[0090] In addition, the overestimation problem is mitigated by delaying Actor updates and adding random exploration entropy terms to control actions.

[0091] In a hierarchical framework, the control layer and the management layer communicate through objectives. The transmission of information achieves decoupling between layers, thereby enabling independent updates and joint control characteristics of the control and management layers, and thus systematically exploring the optimal signal control strategy. The following is a detailed design of the two-layer HSS algorithm in discrete action space: (1) Control layer First, construct the control layer agent Critic target network:

[0092] in, The Critic target network represents the controlling agent. As a discount factor, This represents the reward of the control layer agent at time t. and This represents the target Q-value. After a specific number of steps, based on the empirically replayed buffer data, the target Critic network parameters are used. Update current Critic parameters The updated Critic then guides the update of the policy function.

[0093] To improve the robustness of algorithm updates, this invention uses Critic loss. Huber loss:

[0094] In the above formula, the Huber loss consists of two parts, used to improve the stability of model training. Among them, express Network and target value The loss; similarly, express Network and target value The loss.

[0095] Secondly, regarding the strategy function, this invention is based on the current Critic The expected value and action entropy are used to construct the objective function of the policy, and the policy performance is updated through offline data collection.

[0096] in, Indicates about The loss function of the parameters, express The expectation of the function. Furthermore, For the action entropy term, This is the coefficient of the entropy term.

[0097] Finally, for the control layer policy function The gradient descent algorithm is used to optimize the control layer policy function. parameters Update the policy gradient. The policy gradient containing the entropy term can be expressed as follows:

[0098] in, The policy objective function is expressed as follows: gradient, Representing Q-value pairs under the chain rule gradient, For the policy entropy term with respect to The gradient.

[0099] (2) Management Similar to the control layer update process, the management agent's data is updated based on the experience replay buffer. First, the management agent Critic's target network is defined as follows:

[0100] in, The Critic target network represents the management agent. This represents the reward for the managing agent at time t. and This represents the target Q value. Therefore, a dual-critic loss for management can be constructed:

[0101] Secondly, the management agent's policy function is primarily responsible for issuing goals to the control layer agent. It needs to sample the joint state observations of the region and update them. The management layer policy function is defined as follows:

[0102] Management Strategy The goal is to maximize its corresponding Q value; therefore, the objective function for policy learning is:

[0103] in, Indicates about The loss function of the parameters, express The expectation of the function. Furthermore, For the action entropy term, This is the coefficient of the entropy term.

[0104] Finally, the policy gradient update with entropy term in the management layer is expressed as:

[0105] in, The policy objective function is expressed as follows: gradient, This indicates that the chain rule implements the Q-value pair of the policy parameters. gradient propagation, For the policy entropy term with respect to The gradient.

[0106] Tables 1 and 2 show the agent network structure configurations for the control and management layers, respectively. Both the Actor and Critic maintain their respective target networks. Table 1. Management Agent Network Structure

[0107] Table 2. Control Layer Agent Network Structure

[0108] To verify the method of this invention, experiments were conducted using a 6x6 composite road network, such as... Figure 7The image shows a heatmap of the maximum speed limit for a 6x6 road network. The nodes at the edge of the road network are coded Origin / Destination points. Figure 8 The traffic flow curves for the 6x6 road network are shown in Table 3, where each traffic flow includes 6 pairs of O / D lines. Table 3. 6x6 Road Network Traffic Flow O / D Routes

[0109] The comparison algorithms used were HA2C (Hierarchical Advantage Actor-Critic), MSAC (Multi-agent Soft Actor-Critic), and MA2C (Multi-agent Advantage Actor-Critic): 1) HA2C: Similar to the HSS algorithm framework proposed in this invention, a hierarchical A2C algorithm was designed for comparison and improvement.

[0110] 2) MSAC: This technique combines maximum entropy optimization and reparameterization to achieve efficient and low-variance gradient estimation. Its core idea is to guide the policy towards high-value action distributions through the Q-function while maintaining the policy's exploratory nature. It is one of the mainstream techniques in current deep reinforcement learning.

[0111] 3) MA2C: It adopts a distributed neighbor collaboration mechanism and spatial discount factor to solve the credit allocation problem through local reward weighting.

[0112] like Figures 9-10 As shown, four algorithms—HSS, HA2C, MSAC, and MA2C—were used for training, with each algorithm trained 5 times. The solid line in the figure represents the average reward curve of the 5 experiments, and the shaded area represents the calculated standard deviation. The time step per round was 3600 / 5 = 720 steps, and the total number of training rounds was 4e5 / 720 = 555. It is evident that HSS performed best, with the average reward curve of the control layer agent stabilizing at around 760, and the average reward of the management layer agent stabilizing at around 7000.

[0113] Figure 11 The average speed of the vehicles over the last 10 rounds. Two speed troughs were observed (t=1300s, t=2800s), combined with... Figure 8 It can be seen that the speed trough originated from two traffic flow peaks. All algorithms showed a certain degree of regulation capability when dealing with peak pressure (speed recovery after the trough > 1.5m / s), among which HSS led with its stable advantage throughout, with an average speed of 8.133m / s, which is 43.6% higher than the benchmark MA2C, confirming its dynamic adaptability.

[0114] If traffic flows smoothly overall, the cumulative waiting time and the average waiting time should show a synchronous trend. Figure 14 The study revealed that during the peak traffic flow periods of 1000-1500s and 2500-3000s, all algorithms exhibited peak numbers of stopped vehicles, verifying the positive correlation between high traffic volume and accumulated waiting time. However, further analysis... Figure 12 and Figure 13 An interesting conclusion can be drawn: at the end of the simulation (3250s-3600s), the increase in traffic flow approaches zero. Figure 8 While the cumulative waiting time decreased, the average waiting time for HA2C, MSAC, and MA2C increased against the trend, indicating that their control strategies exhibited a lag effect in traffic density switching scenarios—the high-density conservative strategy failed to adapt to low-flow conditions in a timely manner. In contrast, HSS demonstrated consistent control performance, confirming the robust adaptability of the hierarchical architecture in dynamic traffic state transitions.

[0115] Figure 14 This represents the cumulative number of stops for a 6x6 road network. The corresponding traffic flow curve is shown below. Figure 8 All four algorithms exhibited peak cumulative waiting times between 1000-1500s and 2500-3000s, but the peak amplitude of HSS was significantly lower than that of the other methods (MA2C's second peak reached 150 vehicles, while HSS only reached 57 vehicles). Furthermore, HSS showed smaller fluctuations in waiting time during off-peak hours, demonstrating the best overall control performance.

[0116] like Figure 15 In terms of travel time metrics, the HSS algorithm demonstrates comprehensive performance advantages: its median travel time is 156 seconds, a reduction of 19.6% and 22.0% compared to MA2C (194 seconds) and MSAC (200 seconds), respectively, while its dispersion (IQR=66 seconds) is reduced by 35.3% and 37.7% year-on-year. Although HA2C achieves suboptimal performance through its hierarchical architecture (median 170 seconds, IQR=73 seconds), its travel time volatility is still 10.6% higher than that of HSS. In particular, HSS excels in reliability metrics—75% of its travel times are kept within 187 seconds, providing more stable traffic assurance in dynamic traffic environments.

[0117] Figure 16 The data shows the average number of stops per round, cumulative waiting time, and average vehicle speed changes with the number of training rounds. Considering all metrics, HSS, HA2C, and MSAC all exhibit stable convergence trends in terms of cumulative waiting time. However, MA2C shows significant performance degradation in the later stages of training (around 240 rounds), with its cumulative waiting time increasing instead of decreasing, ultimately failing to achieve balanced control of all metrics. This result suggests that the MA2C algorithm may suffer from incomplete policy learning.

[0118] Figure 17 A scatter plot of the average speed of emergency vehicles is shown, and Table 5 lists the average speed statistics of emergency vehicles within a round. Combined with the data in Table 4, it can be seen that compared to the average speed of civilian vehicles, the average speed of HSS, HA2C, MA2C, and MSAC emergency vehicles increased by 0.044 m / s, 0.313 m / s, 0.433 m / s, and 0.383 m / s, respectively, within a round.

[0119] Table 4. Average Indicators for 6x6 Network Rounds

[0120] Table 5 Speed ​​Indicators for 6x6 Emergency Vehicles

[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0122] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A traffic signal control method based on regional hierarchical multi-agent reinforcement learning, characterized in that, include: The target traffic network is divided into regions, and a management layer agent is configured for each region, while a control layer agent is configured for each intersection within the region. During each control cycle, each control layer agent collects the lane density of its corresponding intersection approach lane and receives the area control target issued by its management layer agent. Each control layer agent dynamically corrects the received area control target based on the lane density of the intersection approach lanes to obtain the corrected control target, and outputs a traffic light phase control action through the control layer strategy function based on the corrected control target, and executes the action. Each control layer agent calculates the control layer reward based on its control effect and stores the experience data in the experience replay buffer. The management layer agent aggregates the state observations of all control layer agents within its jurisdiction to form a regional joint state observation. The management layer agent, based on the joint state observation of the region, outputs regional control objectives through the management layer policy function to guide the control layer, and periodically calculates management layer rewards based on the overall traffic operation effect of the region. Using the data in the experience playback buffer, the control layer policy function and the management layer policy function are updated respectively using an offline policy training method to optimize the overall traffic signal control performance.

2. The method as described in claim 1, characterized in that, The division of the target traffic network into regions specifically includes: Abstract the target traffic network into a graph structure, where nodes represent intersections and edges represent roads; Obtain the geographic coordinates of each intersection; The K-Means clustering algorithm is used to divide all intersections into several regions based on their geographical coordinates.

3. The method as described in claim 1, characterized in that, The formula for dynamic correction is: in, The corrected control target, They are respectively and Lane density of the approach lanes at the intersection at any given time. Indicates the area control objective.

4. The method as described in claim 1, characterized in that, The control layer rewards include rewards for civilian vehicles and rewards for emergency vehicles, calculated using the following formula: in, For control layer intelligent agents The total reward at time t, This indicates a reward for private vehicles. These are the weighting coefficients. Rewards for emergency vehicles, for Enter the control layer intelligent agent at any time The number of emergency vehicles within the controlled area; in, Speed ​​limits for lanes, Minimum driving speed, The average speed of social vehicles. The average speed of the emergency vehicle. For the number of lanes, and These represent the queue lengths at the previous and next time points, respectively. and These represent the lane vehicle density at the preceding and following times, respectively. , These are the weighting coefficients.

5. The method as described in claim 1, characterized in that, The empirical data is as follows: in, Empirical data collected to represent the control agent's policy. For control layer intelligent agents Total reward This represents the regional control target at time t. This represents the regional control target at time t+1. This represents the state observation at time t. This represents the state observation at time t+1.

6. The method as described in claim 1, characterized in that, State observations of the control layer agent include at least one or more of the following: The intersection edge density, the queue length of vehicles in each approach lane of the intersection, the maximum / minimum green light phase, the current traffic light phase, the lane density of the approach lanes of the intersection, the approach lane density difference, the number of emergency vehicles in the approach lanes, the weighted time-varying average speed of emergency vehicles in the approach lanes, the distance of emergency vehicles to the intersection, and the waiting time of emergency vehicles.

7. The method as described in claim 1, characterized in that, The current management agent weights the rewards of other management agents to fully balance the inter-regional collaboration mechanism. The management layer reward is as follows: in, To coordinate the reward coefficients of the current management agent and neighboring agents, Indicates the region A collection of control agents, Represents the control layer intelligent agent The total reward at time t, It is a region The set of adjacent regions.

8. The method as described in claim 1, characterized in that, Control layer agent Critic target network: in, The Critic target network represents the controlling agent. As a discount factor, Represents the control layer intelligent agent The total reward at time t, and Indicates the target Q value. This represents the state observation at time t+1. This represents the area control target at time t+1. Indicates the operation of the traffic light phase control; The management agent Critic target network is defined as follows: in, The Critic target network represents the management agent. This represents the reward for the management agent at time t. and Indicates the target Q value. This represents the joint state observation of the region at time t+1.

Images