Highway hard shoulder management method and system for traffic resilience

Abstract

The present application relates to the technical field of traffic control, in particular to a highway hard shoulder control method and system for traffic resilience, which method comprises collecting multi-dimensional traffic data of the highway; inputting the multi-dimensional traffic data of the highway into a multi-agent deep reinforcement learning model trained, and outputting a decision on whether each hard shoulder of the highway is open at the current time; the multi-dimensional traffic data of the highway comprises traffic flow, vehicle speed and interchange entrance and exit flow; the multi-agent deep reinforcement learning model is trained based on a Markov decision model, and the road sections in the interchange group are coordinated and optimized through a reward function; the Markov decision model represents the decision-making process of the dynamic opening of the hard shoulder; the present application can realize the dynamic opening and coordinated control of the hard shoulder of the high-density interchange group road section, effectively relieve traffic congestion, reduce the risk of operation, improve the traffic capacity while ensuring traffic safety, and has significant engineering application value and broad popularization prospect.

Description

Technical Field

[0001] This invention relates to the field of traffic control technology, and in particular to a method and system for managing hard shoulders of highways with traffic resilience. Background Technology

[0002] In highway operation and management, traffic resilience reflects the road system's ability to maintain stability and recover quickly in the face of traffic flow fluctuations, emergencies, or structural bottlenecks. Enhancing traffic resilience is crucial for alleviating congestion and ensuring operational safety. Currently, highway traffic volume continues to grow, and traffic congestion frequently occurs during morning and evening rush hours and holidays. As key nodes of backbone corridors, the traffic efficiency and operational safety of high-density interchange clusters directly affect the overall service level of highways. Physical expansion requires huge investments and is severely limited by land resources; intelligent expansion has become the main way to improve service levels, and its core technology is intelligent management and control in the scenario of open hard shoulders. At present, most engineering practices for opening hard shoulders involve fixed thresholds and fixed sections, and existing dynamic management methods for hard shoulders are difficult to accurately match with the dynamic fluctuations and spatiotemporal migration characteristics of traffic flow in high-density interchange clusters.

[0003] In current engineering practice, hard shoulder opening control is based on fixed thresholds and fixed sections. With the development of artificial intelligence technology, some studies have introduced intelligent optimization algorithms to achieve dynamic control of hard shoulders. For example, patent CN119851480A discloses a method, equipment, and medium for managing hard shoulders on highways. It acquires traffic flow parameters of highway sections through a nomadic workstation and optimizes the control through predictive model simulation and genetic algorithms, achieving real-time, intelligent, and globally optimal hard shoulder management. Patent CN120580849A discloses a collaborative management decision-making method for dynamic opening of hard shoulders and upstream diversion on highways. It proposes a collaborative management decision-making method for hard shoulder opening and upstream diversion under non-accident high-volume and abnormal event conditions, organically combining dynamic hard shoulder opening with upstream diversion to maximize traffic management benefits. Patent CN120452186A discloses a dynamic opening method for hard shoulders of highways based on NFGD model and simulation platform. Based on NFGD model and simulation platform, it integrates simulation, perception and control modules, and designs intelligent control algorithm and feedback mechanism to realize dynamic opening control of hard shoulders.

[0004] However, existing methods for managing hard shoulders still have the following limitations: ① Existing control strategies are mostly targeted at single road segments, which makes it difficult to fully characterize the traffic flow coupling relationship between upstream and downstream road segments in high-density interchange groups, and lacks systematic modeling and collaborative control research on the linkage effect between multiple road segments and multiple nodes; ② Hard shoulder management often uses fixed threshold methods (such as a speed threshold of 60km / h), which are difficult to accurately match with the dynamic fluctuations and spatiotemporal migration characteristics of traffic flow in high-density interchange sections; ③ Existing control strategies focus on improving traffic efficiency, and safety considerations usually rely on simple rules or single-objective optimization methods, which are difficult to cope with complex traffic environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies in terms of traffic environment adaptability, dynamic decision-making ability and comprehensive safety assurance, and to provide a method and system for highway hard shoulder management oriented towards traffic resilience.

[0006] In a first aspect, the present invention provides a method for managing hard shoulders of highways with traffic resilience, comprising: Collect multi-dimensional traffic data for highways; Input multi-dimensional traffic data from the highway into the trained multi-agent deep reinforcement learning model, and output the decision on whether each hard shoulder of the highway is open at the current moment; The multi-dimensional traffic data of the highway includes traffic flow, vehicle speed and interchange entrance and exit flow. The multi-agent deep reinforcement learning model is trained based on the Markov decision model and uses the reward function to coordinate and optimize the road segments in the interchange group. The reward function includes traffic efficiency reward, traffic safety reward and spatiotemporal consistency reward, and the Markov decision model represents the decision-making process for the dynamic opening of the hard shoulder.

[0007] A complete collaborative management method for hard shoulders of high-density interchange groups is proposed. By learning the reward function, the coupling relationship between upstream and downstream road segments in the interchange group is characterized, and a multi-agent interaction and reward decomposition mechanism is established to achieve coordination of global and local objectives.

[0008] The traffic efficiency reward incorporates total travel time to optimize road network operation efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow, and the time step.

[0009] Preferably, the traffic safety reward is determined based on traffic conflict risk, speed fluctuation, and lane-changing frequency. The risk of traffic conflict is determined by the number of vehicles in the road segment; The speed fluctuation is determined by the number of vehicles and their speed within the road segment; The frequency of lane-changing behavior is determined by the number of times a vehicle changes lanes.

[0010] Preferably, the spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively.

[0011] Preferably, the multi-agent deep reinforcement learning model includes several agents, each agent corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data of its respective segment and outputs a decision on whether to open the hard shoulder corresponding to the segment.

[0012] Preferably, the training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0013] Preferably, the convergence criteria of the multi-agent deep reinforcement learning model include value function convergence and policy convergence; The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0014] Preferably, the multi-agent deep reinforcement learning model includes a global reward decomposition mechanism to decompose the global reward into a basic reward and a gain reward. The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process.

[0015] Preferably, the multi-agent deep reinforcement learning model further includes a priority experience replay strategy; In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0016] In a second aspect, the present invention provides a highway hard shoulder management system for traffic resilience, used to perform the method described in the first aspect, comprising: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models. They are used to generate state-action-result triplets based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by control actions, to represent the decision-making process for the dynamic opening of hard shoulders.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention abstracts the decision-making process of dynamically opening hard shoulders into a Markov decision model, and combines a reward function to characterize the coupling relationship between upstream and downstream road segments in the interchange group; it achieves coordination of global and local objectives based on the characteristics of multi-agent interaction. Attached Figure Description

[0018] Figure 1 This is a flowchart of a highway hard shoulder management method for traffic resilience in one embodiment of the present invention; Figure 2 This is a scene diagram of a high-density interchange complex on a highway, as shown in one embodiment of the present invention. Figure 3 This is a schematic diagram of time coordination reward in one embodiment of the present invention; Figure 4 This is a schematic diagram of spatial coordination reward in one embodiment of the present invention; Figure 5 This is a framework diagram of a collaborative management algorithm for highway hard shoulder based on a multi-agent deep reinforcement learning model in one embodiment of the present invention; Figure 6 This is a structural diagram of a multi-agent deep reinforcement learning model in one embodiment of the present invention; Figure 7 This is a structural diagram of a multi-agent deep reinforcement learning model in another embodiment of the present invention; Figure 8 This is a structural diagram of a multi-agent deep reinforcement learning model in another embodiment of the present invention; Figure 9 This is a collaborative control diagram of the hard shoulder of a high-density interchange group on a highway, as shown in one embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0020] Example 1 Please refer to the example below. Figure 1 and Figure 2 As shown, this embodiment provides a highway hard shoulder management method oriented towards traffic resilience, including: Collect multi-dimensional traffic data for highways; Input multi-dimensional traffic data from the highway into the trained multi-agent deep reinforcement learning model, and output the decision on whether each hard shoulder of the highway is open at the current moment; The multi-dimensional traffic data of the highway includes traffic flow, vehicle speed and interchange entrance and exit flow. The multi-agent deep reinforcement learning model is trained based on the Markov decision model and uses the reward function to coordinate and optimize the road segments in the interchange group. The reward function includes traffic efficiency reward, traffic safety reward and spatiotemporal consistency reward, and the Markov decision model represents the decision-making process for the dynamic opening of the hard shoulder.

[0021] Markov Decision Processes (MDPs) are the foundational model for reinforcement learning, defining elements such as states, actions, and reward functions. Multi-agent deep reinforcement learning (MAL) models are training and decision-making methods implemented on the MDP framework. Specifically, the MDP is established during the modeling phase, and MDL is used for solving or optimizing the algorithm during implementation. While a single MDP is primarily suitable for single-agent or fully observable environments, high-density interchanges are characterized by multiple road segments, highly coupled multi-agent systems, and partially observable environments. A simple MDP cannot directly learn the cooperative strategies between road segments. Introducing MDL enables collaborative decision-making among agents based on MDPs, fully utilizing global information for policy optimization.

[0022] As one possible application, this invention can be applied to high-density interchange clusters on highways, treating interchanges as nodes and the road segments between two nodes as control units, with each control unit corresponding to an intelligent agent. The high-density interchange cluster refers to high-traffic highway sections with an average spacing of less than 3 km and more than 3 consecutive interchanges, typically found in urban agglomeration areas.

[0023] As one possible implementation, the multi-agent deep reinforcement learning model includes several agents, each corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data for its respective segment and outputs a decision on whether to open the hard shoulder corresponding to the segment.

[0024] Various traffic status data in the multi-dimensional traffic data of highways can be collected in real time by roadside sensing devices (such as millimeter-wave radar, lidar, and radar-visual integrated machines), including indicators such as traffic flow, speed, and flow at interchange entrances and exits.

[0025] Example 2 Please refer to the example below. Figure 3 and Figure 4 As shown, this embodiment is a further optimization based on embodiment 1 or 2, providing a reward function to optimize the highway hard shoulder management method.

[0026] As one possible implementation, the traffic efficiency reward is introduced into the total travel time to optimize road network operating efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow, and the time step.

[0027] Specifically, total travel time (TTS) is related to the inflow of traffic flow ( ) and outflow ( Closely related to this, according to traffic network theory, TTS can be represented by the following formula:

[0028] in: This represents the total time step. For time intervals; This represents the total inflow into the road network. This represents the total outflow of the road network. This represents the number of vehicles in the road network at the initial moment.

[0029]

[0030] Therefore, the traffic efficiency bonus can be defined as:

[0031] As one possible implementation, the traffic safety reward is determined based on traffic conflict risk, speed volatility, and lane-changing frequency. The risk of traffic conflict is determined by the number of vehicles in the road segment; The speed fluctuation is determined by the number of vehicles and their speed within the road segment; The frequency of lane-changing behavior is determined by the number of lanes.

[0032] Specifically, in scenarios involving high-density interchanges on highways, the opening of the hard shoulder may lead to fluctuations in traffic flow, potentially triggering traffic conflicts or accidents. To fully consider traffic safety during the decision-making process, this embodiment designs a traffic safety reward function with traffic conflict risk (TTC), speed volatility, and lane-changing frequency as core indicators:

[0033] in: , , These are the weights of the reward function for TTC, speed volatility, and lane-changing behavior frequency, respectively. For a moment Number of vehicles within the road section; It is a constant used to avoid It is 0 (usually taken as 0.1); The threshold for TTC determination; This represents the total number of lane changes for vehicles within the current time period. (.) is an indicator function. This represents the average speed of vehicles on that road segment at that moment. Let be the speed of the i-th vehicle.

[0034] As one possible implementation, the spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively.

[0035] Specifically, overly frequent or spatially inconsistent hard shoulder opening and closing operations can lead to traffic flow instability, resulting in traffic fluctuations or bottlenecks, and even increasing the risk of traffic accidents. To ensure the stability and operability of the strategy, this embodiment introduces a spatiotemporal consistency reward function. This reward function optimizes the process from both temporal and spatial dimensions, aiming to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments. This ensures the continuity and coordination of hard shoulder opening and closing operations during the decision-making process, thereby improving the overall stability and efficiency of traffic flow and reducing unnecessary fluctuations. The coordination and consistency reward function is as follows:

[0036] in: For time-consistent reward functions; For spatial consistency reward function; , The weights for the time and space coordination reward functions are 0.6 and 0.4, respectively, and are preferably taken as 0.6 and 0.4.

[0037] like Figure 3As shown, to study the changes in road traffic conditions over consecutive time intervals, this paper sets each time interval to 10 minutes and designs a reward function based on this. A hard shoulder switching decision is made every 10 minutes. To ensure temporal continuity in the decision-making process, the total number of changes in the opening and closing states of all hard shoulders in the controlled road segment between two adjacent decisions is defined as the number of transitions. The control objective is to minimize the number of transitions, thereby maintaining decision stability and system temporal coordination. The temporal coordination reward function is as follows:

[0038] in, For the first The first time period The hard shoulder of each road section is open; For the first The first time period The hard shoulder of each road section is open; A time jump is counted as one time jump from 0 to 1 and from 1 to 0.

[0039] like Figure 4 As shown, to avoid the hard shoulder opening and closing statuses being too dispersed across different road sections, the number of lane changes from the main road to the hard shoulder and from the hard shoulder to the main road is increased. Therefore, a spatial coordination reward function is added to the hard shoulder opening and closing strategy. The controlled road sections are divided into... Each section of road For open lane components, consecutive road segments that are spatially connected and have open hard shoulders within the same 10-minute time interval are counted as one open lane component. The spatial coordination reward function is as follows:

[0040] As a preferred implementation, considering traffic efficiency, traffic safety, and spatiotemporal consistency, a combined reward function is provided:

[0041] in, For traffic efficiency reward function, For traffic safety reward functions, For a spatiotemporally consistent reward function, , , The weights for the reward functions are traffic efficiency, traffic safety, and spatiotemporal consistency, respectively, and can preferably be 0.5, 0.3, and 0.2.

[0042] Example 3 Please refer to the example below. Figure 5 and Figure 6As shown in the figure, this embodiment is a further optimization based on embodiment 2, providing a multi-agent deep reinforcement learning model and a framework diagram of a collaborative management algorithm for highway hard shoulder based on the multi-agent deep reinforcement learning model.

[0043] in, Figure 6 middle Let be the observation sequence of the nth agent.

[0044] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0045] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0046] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. When determining convergence, the multi-agent deep reinforcement learning model needs to satisfy both value function convergence and policy convergence simultaneously.

[0047] The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0048] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0049] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0050] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0051] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0052] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0053] in: The value of the global reward function. To set a threshold.

[0054] Example 4 Please refer to the example below. Figure 5 and Figure 7 As shown in the figure, this embodiment further optimizes the previous embodiment and provides another multi-agent deep reinforcement learning model that introduces a global reward decomposition mechanism, as well as the framework diagram of the collaborative management algorithm for highway hard shoulder based on this multi-agent deep reinforcement learning model.

[0055] in, Figure 7 middle Let be the observation sequence of the nth agent.

[0056] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0057] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0058] As a preferred embodiment, this method introduces a global reward decomposition mechanism to decompose the global reward. Decomposed into basic rewards and bonus By allocating different weights, the balance between local and global objectives can be optimized; The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process. The specific formula is as follows:

[0059]

[0060]

[0061] in, , where is the weight learned through the hybrid network, and is the basic reward. Rewards are distributed among the agents primarily through a hybrid network. The basic reward for each agent is... Allocation is based on their contribution to local traffic optimization. Specifically, for intelligent agents... The basic reward it receives can be expressed as:

[0062] in, These are the weights learned through the network.

[0063] Bonus This is used to incentivize agents that contribute significantly to the global optimization process. The gain reward for each agent... The agent's reward is allocated based on its contribution to overall traffic flow optimization. The agent's gain reward is proportional to its contribution, as shown in the following formula:

[0064] in: For intelligent agents Contribution to the overall goal, The total contribution of all intelligent agents.

[0065] In each training iteration, the agent is based on the total reward received. Update its strategy The total reward guides the agent to optimize its decisions, enabling it to strike a balance between local control and global optimization, thereby improving the control effect of the hard shoulder.

[0066] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation value is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0067] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0068] in, The preset convergence threshold can also indicate that the Q function tends to converge. The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0069] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered converged.

[0070] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0071] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0072] in: The value of the global reward function. To set a threshold.

[0073] Example 5 Please refer to the example below. Figure 5 and Figure 8 As shown in the figure, this embodiment further optimizes upon embodiment 1 or 2, providing another multi-agent deep reinforcement learning model that introduces a priority experience replay strategy and its framework diagram of a collaborative management algorithm for highway hard shoulder based on this multi-agent deep reinforcement learning model.

[0074] in, Figure 8 middle Let be the observation sequence of the nth agent.

[0075] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0076] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back and updated, and the agent's strategy is updated accordingly. Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0077] As a preferred approach, this embodiment introduces a priority experience replay strategy. It measures the importance of samples by calculating the temporal difference (TD) error and assigns them priority, thereby increasing the probability of key experiences being selected during the sampling process. This method can significantly improve the learning efficiency and convergence speed of the agent.

[0078] In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0079] As one possible implementation, multiple agents share an experience pool to fully utilize data generated from interactions between different agents and the environment, promoting collaborative learning. Agents evaluate the importance of each experience using TD error and prioritize experiences based on the absolute value of this error. Specifically, for a state transition triple... ,in, The TD error calculation formula is as follows: This formula represents the "reward" from environmental feedback after each action.

[0080] Based on the TD error, for each empirical sample Assign a priority:

[0081] in, This is a small constant used to prevent the priority from being zero. In this way, experiences with larger TD errors are given higher priority and are thus extracted more often during playback.

[0082] During training, the agent does not draw samples uniformly from the experience pool, but rather performs weighted sampling based on priority. Higher-priority experience samples are sampled more frequently, while lower-priority samples are sampled less often. The specific sampling process is based on an importance sampling method, and is performed using the following formula:

[0083] in: For the first Priority of each empirical sample; This is a hyperparameter that controls the degree of influence of priority. When, it is equivalent to uniform sampling; when At that time, samples with higher priority are more likely to be sampled.

[0084] After each experience sample is drawn from the experience pool, the algorithm trains the Q-network based on these samples. The agent's Q-value function is updated using TD error and priority.

[0085] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that the difference between the agent's Q-value function and the Bellman expectation value is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0086] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0087] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the change in the distribution of action choices during the iteration process is less than the policy convergence threshold.

[0088] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0089] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0090] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0091] in: The value of the global reward function. To set a threshold.

[0092] Example 6 Please refer to the example below. Figure 9 As shown, this embodiment further optimizes the previous embodiment by providing a multi-agent reinforcement learning model that simultaneously introduces a global reward decomposition mechanism and a priority experience replay strategy. Based on this model, this embodiment explains the principle of highway hard shoulder management using a high-density interchange group on a highway.

[0093] in, Figure 9 middle Let be the observation sequence of the nth agent.

[0094] Traffic Condition Perception and Modeling: High-density interchange clusters refer to high-volume highway sections with an average spacing of less than 3km and more than 3 consecutive interchanges, typically found in urban agglomerations. This invention targets high-density interchange cluster scenarios on highways, treating interchanges as nodes and the road segments between two nodes as control units. Various traffic condition data can be collected in real time by roadside sensing devices (such as millimeter-wave radar, lidar, and integrated radar-visual systems), including indicators such as traffic flow, speed, and interchange entrance / exit traffic. Each road segment... The state in the control cycle Internally, it can be described as:

[0095] in: For road section Traffic flow in each lane; For road section Upstream interchange Traffic flow at the entrance; For road section Downstream interchange Traffic flow at exits; For road section Average vehicle speed in each lane; The hard shoulder of the current road section is open; The hard shoulder status is from the previous control cycle; Traffic flow ratio between main road and hard shoulder Let M be the dimension of the state space, and M be the space of real numbers.

[0096] Markov Decision Model Construction: In high-density interchange clusters on highways, the dynamic opening decision of hard shoulders is a complex multi-factor optimization problem. This invention models it as a Markov decision model, defining a reward that integrates traffic efficiency, traffic safety, and spatiotemporal consistency to achieve the optimal decision strategy. Based on this model, a multi-agent deep reinforcement learning method is used for training to collaboratively optimize the hard shoulder management method, improving overall traffic efficiency and safety. This includes the following sub-steps: A. State Compression: High-density interchange groups include Each section of road ( , , … Each sub-segment can have its hard shoulder opened / closed independently. Traffic conditions for each segment are as follows: Each traffic characteristic dimension describes the traffic flow (speed, density, traffic volume, openness, etc.). Define the first... road section At any moment The status is:

[0097] The state matrix for the entire road segment is as follows:

[0098] Convert it to a row vector to obtain the original state vector:

[0099] Since directly using high-dimensional states leads to the curse of dimensionality and excessive computational complexity, Principal Component Analysis (PCA) is used to reduce the dimensionality of the state space. PCA reduces the dimensionality of the state space by analyzing the covariance matrix of historical traffic state data. Perform eigenvalue decomposition to extract the eigenvectors along the direction of maximum variance in the data, and construct a mapping matrix:

[0100] in: It is the eigenvector corresponding to the largest eigenvalue. This represents the dimension of the state space after dimensionality reduction.

[0101] Through the mapping matrix high-dimensional state Projected onto the low-dimensional state vector:

[0102] A low-dimensional state space is defined as:

[0103] B. Action Space Definition: The control strategy is manifested in whether the hard shoulder is enabled on each road segment. Therefore, each road segment corresponds to a binary action variable, indicating whether the hard shoulder is enabled on that segment at the current moment, thus defining the action space. The set of hard shoulder controls for each road segment:

[0104] in: Indicates the first The hard shoulder of this section of road is open at that moment; This indicates that the hard shoulder is open at that moment.

[0105] C. State transition modeling Traffic flow in high-density interchange sections of highways is highly dynamic and uncertain. Its state evolution is affected by multiple factors such as traffic volume, vehicle speed distribution, traffic interference at entrances and exits, and unforeseen events, making the state transition function... It is difficult to accurately describe this using analytical models. Therefore, this invention employs an empirical modeling method based on measured multi-source traffic data. It utilizes traffic operation data collected by roadside sensing devices (such as millimeter-wave radar, lidar, and integrated radar-visual systems) and traffic monitoring systems to construct an approximate state transition function. The specific steps are as follows: ① Data collection: Relying on the roadside sensing and traffic monitoring system, real-time collection of multi-dimensional traffic parameters such as road segment flow, average vehicle speed, density, and entrance / exit flow is carried out; ② Input control action: at time Record traffic conditions and the corresponding control actions of the control strategy. (Hard shoulder opening / closing status), and continuously monitor its impact on traffic flow; ③ State transition observation: during action Under its influence, the traffic status at the next moment is observed and recorded. Forming a state-action-result triple. ; ④ Constructing an empirical dataset: By continuously collecting traffic operation data across multiple time periods and scenarios, a large number of state transition samples are accumulated, and then the state transition distribution is approximated and modeled within a reinforcement learning framework.

[0106] This invention, based on an embodiment, uses a reinforcement learning framework to model the dynamic management process of hard shoulders as a multi-agent problem, where each control unit is considered an agent. Each agent aims to maximize the cumulative reward of its control strategy. The state space and action space correspond to different traffic states and hard shoulder control strategies, respectively, and the immediate benefits of actions in a specific state are evaluated using the reward function described in Embodiment 2.

[0107] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action, then select the action with the maximum value as the current optimal action; Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0108] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from the observation data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back and updated, and the agent's strategy is updated accordingly. Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0109] As a preferred solution, this embodiment introduces a global reward decomposition mechanism and a priority experience replay strategy; The global reward decomposition mechanism is used to decompose global rewards. Decomposed into basic rewards and bonus By allocating different weights, the balance between local and global objectives can be optimized; The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process. The specific formula is as follows:

[0110]

[0111]

[0112] in, , where is the weight learned through the hybrid network, and is the basic reward. Rewards are distributed among the agents primarily through a hybrid network. The basic reward for each agent is... Allocation is based on their contribution to local traffic optimization. Specifically, for intelligent agents... The basic reward it receives can be expressed as:

[0113] in, These are the weights learned through the network.

[0114] Bonus This is used to incentivize agents that contribute significantly to the global optimization process. The gain reward for each agent is calculated as follows: The agent's reward is allocated based on its contribution to overall traffic flow optimization. The agent's gain reward is proportional to its contribution, as shown in the following formula:

[0115] in: For intelligent agents Contribution to the overall goal, The total contribution of all intelligent agents.

[0116] In each training iteration, the agent is based on the total reward received. Update its strategy The total reward guides the agent to optimize its decisions, enabling it to strike a balance between local control and global optimization, thereby improving the control effect of the hard shoulder.

[0117] The priority experience replay strategy measures the importance of samples by calculating temporal difference (TD) error and assigns them priority, thereby increasing the probability of key experiences being selected during the sampling process. This method can significantly improve the learning efficiency and convergence speed of the agent.

[0118] In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0119] As one possible implementation, multiple agents share an experience pool to fully utilize data generated from interactions between different agents and the environment, promoting collaborative learning. Agents evaluate the importance of each experience using TD error and prioritize experiences based on the absolute value of this error. Specifically, for a state transition triple... The formula for calculating the TD error is as follows:

[0120] Based on the TD error, for each empirical sample Assign a priority:

[0121] in, This is a small constant used to prevent the priority from being zero. In this way, experiences with larger TD errors are given higher priority and are thus extracted more often during playback.

[0122] During training, the agent does not draw samples uniformly from the experience pool, but rather performs weighted sampling based on priority. Higher-priority experience samples are sampled more frequently, while lower-priority samples are sampled less often. The specific sampling process is based on an importance sampling method, and is performed using the following formula:

[0123] in: For the first Priority of each empirical sample; This is a hyperparameter that controls the degree of influence of priority. When, it is equivalent to uniform sampling; when At that time, samples with higher priority are more likely to be sampled.

[0124] After each experience sample is drawn from the experience pool, the algorithm trains the Q-network based on these samples. The agent's Q-value function is updated using TD error and priority.

[0125] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that the difference between the agent's Q-value function and the Bellman expectation value is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0126] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0127] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the change in the distribution of action choices during the iteration process is less than the policy convergence threshold.

[0128] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0129] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0130] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0131] in: The value of the global reward function. To set a threshold.

[0132] Example 7 This embodiment provides a highway hard shoulder management system for traffic resilience, used to execute the method described in any one of embodiments 1-6, including: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models. They are used to generate state-action-result triplets based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by control actions, to represent the decision-making process of dynamically opening hard shoulders.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing the hard shoulder of highways based on traffic resilience, characterized in that: include: Collect multi-dimensional traffic data for highways; Input multi-dimensional traffic data from the highway into the trained multi-agent deep reinforcement learning model, and output the decision on whether each hard shoulder of the highway is open at the current moment; The multi-dimensional traffic data of the expressway includes traffic flow, vehicle speed, and interchange entrance and exit flow. The multi-agent deep reinforcement learning model is trained based on the Markov decision model and uses a reward function to coordinate and optimize the road segments in the interchange group. High-density interchange group refers to high-volume expressway sections with an average spacing of less than 3km and more than 3 consecutive interchanges. The reward function includes traffic efficiency reward, traffic safety reward and spatiotemporal consistency reward, and the Markov decision model represents the decision-making process for the dynamic opening of the hard shoulder; The spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively. The multi-agent deep reinforcement learning model includes several agents, each agent corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data for its respective segment and outputs the decision on whether to open the hard shoulder corresponding to the segment. The time coordination reward function corresponding to the time coordination reward is: ； in, The hard shoulder of the i-th road segment is open during the t-th time period; The hard shoulder is open in the i-th road segment during the (t+1)-th time period; A change from 0 to 1 and a change from 1 to 0 counts as one time jump. The spatial coordination reward function corresponding to the spatial coordination reward is: ； in, For open lane components; The multi-agent deep reinforcement learning model includes a global reward decomposition mechanism to decompose the global reward into a basic reward and a gain reward. The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process. ； ； ； in, The basic reward is the weight learned through the hybrid network. Rewards are distributed to individual agents through a hybrid network; each agent's basic reward is allocated based on its contribution to local traffic optimization. The agent's gain reward is proportional to its contribution, as shown in the following formula: ； in: Let i be the contribution of agent i to the global objective. The total contribution of all intelligent agents; In each training iteration, the agent updates its policy based on the total reward received; The multi-agent deep reinforcement learning model also includes a priority experience replay strategy; In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

2. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The traffic efficiency reward incorporates total travel time to optimize road network operation efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow, and the time step.

3. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The traffic safety reward is determined based on traffic conflict risk, speed fluctuations, and the frequency of lane-changing behavior. The risk of traffic conflict is determined by the number of vehicles in the road segment; The speed fluctuation is determined by the number of vehicles and their speed within the road segment; The frequency of lane-changing behavior is determined by the number of times a vehicle changes lanes.

4. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The training process of the multi-agent deep reinforcement learning model includes: The state generated by the Markov decision model action The resulting triples fill the experience pool and initialize the value functions and policies of all agents; For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

5. The highway hard shoulder management method based on traffic resilience according to claim 4, characterized in that, The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

6. A highway hard shoulder management system for traffic resilience, characterized in that, For executing claim 1 The method described in any one of the following five categories includes: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models, used to generate states based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by these actions. action The resulting triples characterize the decision-making process for the dynamic opening of the hard shoulder.

Images