Highway hard shoulder management and control method and system oriented to traffic toughness

Abstract

The invention relates to the technical field of traffic control, in particular to a highway hard shoulder control method and system oriented to traffic toughness, and the method comprises the steps: collecting the multidimensional traffic data of a highway; inputting the highway multi-dimensional traffic data into the trained multi-agent deep reinforcement learning model, and outputting a decision on whether each hard road shoulder of the highway is opened at the current moment; the highway multi-dimensional traffic data comprises traffic flow, vehicle speed and interchange entrance and exit flow, the multi-agent deep reinforcement learning model is trained based on a Markov decision model, and coordinated optimization is performed on road sections in an interchange group through a reward function; according to the method, the hard shoulder dynamic opening and collaborative control of the high-density interchange group road section can be realized, the traffic jam is effectively relieved, the operation risk is reduced, the traffic safety is ensured while the traffic capacity is improved, and the method is suitable for popularization and application. And the method has a remarkable engineering application value and a wide popularization prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic control, in particular to a highway hard shoulder control method and system for traffic resilience. BACKGROUND

[0002] In the operation and management of highways, traffic resilience reflects the ability of the road system to maintain stability and quickly recover in the face of traffic flow fluctuations, emergencies or structural bottlenecks. Improving traffic resilience is particularly critical for relieving congestion and ensuring safe operation. Currently, highway traffic volume is continuously increasing, and traffic congestion on highways often occurs during morning and evening peak hours and holidays. High-density interchange groups, as the main nodes of the backbone channel, directly affect the overall service level of the highway in terms of traffic efficiency and operational safety. The physical expansion mode requires huge investment and is severely limited by land resources. Intelligent expansion has become the main way to improve service levels, and the core technology is intelligent control in the hard shoulder opening scenario. At present, the engineering practice of hard shoulder opening is mostly fixed threshold and fixed section opening, and the existing dynamic control method of hard shoulder is difficult to accurately match the dynamic fluctuations of traffic flow and the characteristics of space-time migration of high-density interchange groups.

[0003] In existing engineering practice, hard shoulder opening control is fixed threshold and fixed section opening. With the development of artificial intelligence technology, some research has introduced intelligent optimization algorithms to achieve dynamic control of hard shoulders. For example, patent CN119851480A discloses a highway hard shoulder control method, device and medium, which obtains traffic flow parameters of a highway section through a nomadic workstation, optimizes through a prediction model and a genetic algorithm, and realizes real-time, intelligent and global optimization of hard shoulder control. Patent CN120580849A discloses a highway hard shoulder dynamic opening and upstream shunting collaborative control decision method, which proposes a control decision method for hard shoulder opening and upstream shunting collaboration under non-accident high flow and abnormal event conditions, organically combines hard shoulder dynamic opening and upstream shunting, and maximizes the efficiency of traffic control. Patent CN120452186A discloses a highway hard shoulder dynamic opening method based on an NFGD model and a simulation platform, which designs intelligent control algorithms and feedback mechanisms based on the fusion of simulation, perception and control modules, and realizes dynamic opening control of hard shoulders.

[0004] However, the existing hard shoulder control method still has the following limitations: ① Most existing control strategies are for a single section, which is difficult to fully depict the traffic flow coupling relationship between upstream and downstream sections in high-density interchange groups, and lacks systematic modeling and collaborative control research on the linkage effect between multiple sections and nodes. ②Hard shoulder control methods usually adopt fixed threshold methods (such as a vehicle speed threshold of 60 km / h), which are difficult to accurately match the dynamic fluctuations and space-time migration characteristics of traffic flow of high-density interchange group road sections; ③Existing control strategies focus on improving traffic efficiency, and the consideration of safety usually relies on simple rules or single objective optimization methods, which are difficult to cope with complex traffic environments. SUMMARY

[0005] The present application aims to overcome the deficiencies of the prior art in traffic environment adaptability, dynamic decision-making ability and comprehensive safety protection, and provides a hard shoulder control method and system for traffic resilience of expressways.

[0006] In a first aspect, the present application provides a hard shoulder control method for traffic resilience of expressways, comprising: collecting multi-dimensional traffic data of the expressway; inputting the multi-dimensional traffic data of the expressway into a trained multi-agent deep reinforcement learning model to output a decision on whether each hard shoulder of the expressway is open at the current time; The multi-dimensional traffic data of the expressway includes traffic flow, vehicle speed and interchange entrance and exit flow, the multi-agent deep reinforcement learning model is trained based on a Markov decision model, and the interchange group is coordinated and optimized by a reward function; The reward function includes traffic efficiency reward, traffic safety reward and space-time consistency reward, and the Markov decision model represents the decision process of dynamic opening of the hard shoulder.

[0007] A complete hard shoulder coordination and control method for high-density interchanges is proposed, which learns to depict the coupling relationship between upstream and downstream road sections in the interchange group through a reward function, establishes a multi-agent interaction and reward decomposition mechanism, and realizes the coordination of global and local objectives.

[0008] The traffic efficiency reward introduces total travel time optimization to improve network operation efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow and the time step.

[0009] Preferably, the traffic safety reward is determined by traffic conflict risk, speed volatility and lane changing frequency; The traffic conflict risk is determined by the number of vehicles on the road section; The speed volatility is determined by the number of vehicles on the road section and the vehicle speed; The lane changing frequency is determined by the number of lane changes of the vehicle.

[0010] Preferably, the spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively.

[0011] Preferably, the multi-agent deep reinforcement learning model includes several agents, each agent corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data of its respective segment and outputs a decision on whether to open the hard shoulder corresponding to the segment.

[0012] Preferably, the training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0013] Preferably, the convergence criteria of the multi-agent deep reinforcement learning model include value function convergence and policy convergence; The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0014] Preferably, the multi-agent deep reinforcement learning model includes a global reward decomposition mechanism to decompose the global reward into a basic reward and a gain reward. The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process.

[0015] Preferably, the multi-agent deep reinforcement learning model further includes a priority experience replay strategy; In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0016] In a second aspect, the present invention provides a highway hard shoulder management system for traffic resilience, used to perform the method described in the first aspect, comprising: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models. They are used to generate state-action-result triplets based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by control actions, to represent the decision-making process for the dynamic opening of hard shoulders.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention abstracts the decision-making process of dynamically opening hard shoulders into a Markov decision model, and combines a reward function to characterize the coupling relationship between upstream and downstream road segments in an interchange group; it achieves coordination of global and local objectives based on the characteristics of multi-agent interaction. Attached Figure Description

[0018] Figure 1 This is a flowchart of a highway hard shoulder management method for traffic resilience in one embodiment of the present invention; Figure 2 This is a scene diagram of a high-density interchange complex on a highway, as shown in one embodiment of the present invention. Figure 3 This is a schematic diagram of time coordination reward in one embodiment of the present invention; Figure 4 This is a schematic diagram of spatial coordination reward in one embodiment of the present invention; Figure 5 This is a framework diagram of a collaborative management algorithm for highway hard shoulder based on a multi-agent deep reinforcement learning model in one embodiment of the present invention; Figure 6 This is a structural diagram of a multi-agent deep reinforcement learning model in one embodiment of the present invention; Figure 7 This is a structural diagram of a multi-agent deep reinforcement learning model in another embodiment of the present invention; Figure 8 This is a structural diagram of a multi-agent deep reinforcement learning model in another embodiment of the present invention; Figure 9 This is a collaborative control diagram of the hard shoulder of a high-density interchange group on a highway, as shown in one embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0020] Example 1 Please refer to the example below. Figure 1 and Figure 2 As shown, this embodiment provides a highway hard shoulder management method oriented towards traffic resilience, including: Collect multi-dimensional traffic data for highways; Input multi-dimensional traffic data from the highway into the trained multi-agent deep reinforcement learning model, and output the decision on whether each hard shoulder of the highway is open at the current moment; The multi-dimensional traffic data of the highway includes traffic flow, vehicle speed and interchange entrance and exit flow. The multi-agent deep reinforcement learning model is trained based on the Markov decision model and uses the reward function to coordinate and optimize the road segments in the interchange group. The reward function includes traffic efficiency reward, traffic safety reward and spatiotemporal consistency reward, and the Markov decision model represents the decision-making process for the dynamic opening of the hard shoulder.

[0021] Markov Decision Processes (MDPs) are the foundational model for reinforcement learning, defining elements such as states, actions, and reward functions. Multi-agent deep reinforcement learning (MAL) models are training and decision-making methods implemented on the MDP framework. Specifically, the MDP is established during the modeling phase, and MDL is used for solving or optimizing the algorithm during implementation. While a single MDP is primarily suitable for single-agent or fully observable environments, high-density interchanges are characterized by multiple road segments, highly coupled multi-agent systems, and partially observable environments. A simple MDP cannot directly learn the cooperative strategies between road segments. Introducing MDL enables collaborative decision-making among agents based on MDPs, fully utilizing global information for policy optimization.

[0022] As one possible application, this invention can be applied to high-density interchange clusters on highways, treating interchanges as nodes and the road segments between two nodes as control units, with each control unit corresponding to an intelligent agent. The high-density interchange cluster refers to high-traffic highway sections with an average spacing of less than 3 km and more than 3 consecutive interchanges, typically found in urban agglomeration areas.

[0023] As one possible implementation, the multi-agent deep reinforcement learning model includes several agents, each corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data for its respective segment and outputs a decision on whether to open the hard shoulder corresponding to the segment.

[0024] Various traffic status data in the multi-dimensional traffic data of highways can be collected in real time by roadside sensing devices (such as millimeter-wave radar, lidar, and radar-visual integrated machines), including indicators such as traffic flow, speed, and flow at interchange entrances and exits.

[0025] Example 2 Please refer to the example below. Figure 3 and Figure 4 As shown, this embodiment is a further optimization based on embodiment 1 or 2, providing a reward function to optimize the highway hard shoulder management method.

[0026] As one possible implementation, the traffic efficiency reward is introduced into the total travel time to optimize road network operating efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow, and the time step.

[0027] Specifically, total travel time (TTS) is related to the inflow of traffic flow. ) and outflow ( Closely related to this, according to traffic network theory, TTS can be represented by the following formula:

[0028] in: This represents the total time step. For time intervals; This represents the total inflow into the road network. This represents the total outflow of the road network. This represents the number of vehicles in the road network at the initial moment.

[0029]

[0030] Therefore, the traffic efficiency reward can be defined as:

[0031] As one possible implementation, the traffic safety reward is determined based on traffic conflict risk, speed volatility, and lane-changing frequency. The risk of traffic conflict is determined by the number of vehicles in the road segment; The speed fluctuation is determined by the number of vehicles and their speed within the road segment; The frequency of lane-changing behavior is determined by the number of lanes.

[0032] Specifically, in scenarios involving high-density interchanges on highways, the opening of the hard shoulder may lead to fluctuations in traffic flow, potentially triggering traffic conflicts or accidents. To fully consider traffic safety during the decision-making process, this embodiment designs a traffic safety reward function with traffic conflict risk (TTC), speed volatility, and lane-changing frequency as core indicators:

[0033] in: , , These are the weights of the reward function for TTC, speed volatility, and lane-changing behavior frequency, respectively. For a moment Number of vehicles within the road section; It is a constant used to avoid It is 0 (usually taken as 0.1); The threshold for TTC determination; This represents the total number of lane changes for vehicles within the current time period. (.) is an indicator function. This represents the average speed of vehicles on that road segment at that moment. Let be the speed of the i-th vehicle.

[0034] As one possible implementation, the spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively.

[0035] Specifically, overly frequent or spatially inconsistent hard shoulder opening and closing operations can lead to traffic flow instability, resulting in traffic fluctuations or bottlenecks, and even increasing the risk of traffic accidents. To ensure the stability and operability of the strategy, this embodiment introduces a spatiotemporal consistency reward function. This reward function optimizes the process from both temporal and spatial dimensions, aiming to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments. This ensures the continuity and coordination of hard shoulder opening and closing operations during the decision-making process, thereby improving the overall stability and efficiency of traffic flow and reducing unnecessary fluctuations. The coordination and consistency reward function is as follows:

[0036] in: For time-consistent reward functions; For spatial consistency reward function; , The weights for the time and space coordination reward functions are 0.6 and 0.4, respectively, and are preferably taken as 0.6 and 0.4.

[0037] like Figure 3As shown, to study the changes in road traffic conditions over consecutive time intervals, this paper sets each time interval to 10 minutes and designs a reward function based on this. A hard shoulder switching decision is made every 10 minutes. To ensure temporal continuity in the decision-making process, the total number of changes in the opening and closing states of all hard shoulders in the controlled road segment between two adjacent decisions is defined as the number of transitions. The control objective is to minimize the number of transitions, thereby maintaining decision stability and system temporal coordination. The temporal coordination reward function is as follows:

[0038] in, For the first The first time period The hard shoulders of each road section are open; For the first The first time period The hard shoulders of each road section are open; A change from 0 to 1 and a change from 1 to 0 counts as one time jump.

[0039] like Figure 4 As shown, to avoid the hard shoulder opening and closing statuses being too dispersed across different road sections, the number of lane changes from the main road to the hard shoulder and from the hard shoulder to the main road is increased. Therefore, a spatial coordination reward function is added to the hard shoulder opening and closing strategy. The controlled road sections are divided into... Each section of road The open lane component is defined as follows: Within the same 10-minute time interval, consecutive road segments that are spatially connected and have open hard shoulders are counted as one open lane component. The spatial coordination reward function is then as follows:

[0040] As a preferred implementation method, considering traffic efficiency, traffic safety, and spatiotemporal consistency, a combined reward function is provided:

[0041] in, For traffic efficiency reward function, For traffic safety reward functions, For a spatiotemporally consistent reward function, , , The weights for the reward functions are traffic efficiency, traffic safety, and spatiotemporal consistency, respectively, and can preferably be 0.5, 0.3, and 0.2.

[0042] Example 3 Please refer to the example below. Figure 5 and Figure 6As shown in the figure, this embodiment is a further optimization based on embodiment 2, providing a multi-agent deep reinforcement learning model and a framework diagram of a collaborative management algorithm for highway hard shoulder based on the multi-agent deep reinforcement learning model.

[0043] in, Figure 6 middle Let be the observation sequence of the nth agent.

[0044] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0045] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0046] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. When determining convergence, the multi-agent deep reinforcement learning model needs to satisfy both value function convergence and policy convergence simultaneously.

[0047] The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0048] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0049] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0050] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0051] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0052] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0053] in: The value of the global reward function. To set a threshold.

[0054] Example 4 Please refer to the example below. Figure 5 and Figure 7 As shown in the figure, this embodiment further optimizes the previous embodiment and provides another multi-agent deep reinforcement learning model that introduces a global reward decomposition mechanism, as well as the framework diagram of the collaborative management algorithm for highway hard shoulder based on this multi-agent deep reinforcement learning model.

[0055] in, Figure 7 middle Let be the observation sequence of the nth agent.

[0056] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0057] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0058] As a preferred embodiment, this method introduces a global reward decomposition mechanism to decompose the global reward. Decomposed into basic rewards and bonus By allocating different weights, the balance between local and global objectives can be optimized; The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process. The specific formula is as follows:

[0059]

[0060]

[0061] in, , where is the weight learned through the hybrid network, and is the basic reward. Rewards are distributed among the agents primarily through a hybrid network. The basic reward for each agent is... Allocation is based on their contribution to local traffic optimization. Specifically, for intelligent agents... The basic reward it receives can be expressed as:

[0062] in, These are the weights learned through the network.

[0063] Bonus This is used to incentivize agents that contribute significantly to the global optimization process. The gain reward for each agent... The agent's reward is allocated based on its contribution to overall traffic flow optimization. The agent's gain reward is proportional to its contribution, as shown in the following formula:

[0064] in: For intelligent agents Contribution to the overall goal, The total contribution of all intelligent agents.

[0065] In each training iteration, the agent is based on the total reward received. Update its strategy The total reward guides the agent to optimize its decisions, enabling it to strike a balance between local control and global optimization, thereby improving the control effect of the hard shoulder.

[0066] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0067] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0068] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

[0069] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0070] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0071] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0072] in: The value of the global reward function. To set a threshold.

[0073] Example 5 Please refer to the example below. Figure 5 and Figure 8 As shown in the figure, this embodiment further optimizes upon embodiment 1 or 2, providing another multi-agent deep reinforcement learning model that introduces a priority experience replay strategy and its framework diagram of a collaborative management algorithm for highway hard shoulder based on this multi-agent deep reinforcement learning model.

[0074] in, Figure 8 middle Let be the observation sequence of the nth agent.

[0075] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0076] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0077] As a preferred approach, this embodiment introduces a priority experience replay strategy. It measures the importance of samples by calculating the temporal difference (TD) error and assigns them priority, thereby increasing the probability of key experiences being selected during the sampling process. This method can significantly improve the learning efficiency and convergence speed of the agent.

[0078] In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0079] As one possible implementation, multiple agents share an experience pool to fully utilize data generated from interactions between different agents and the environment, promoting collaborative learning. Agents evaluate the importance of each experience using TD error and prioritize experiences based on the absolute value of this error. Specifically, for a state transition triple... ,in, The TD error calculation formula is as follows: This formula represents the "reward" from environmental feedback after each action.

[0080] Based on the TD error, for each empirical sample Assign a priority:

[0081] in, This is a small constant used to prevent the priority from being zero. In this way, experiences with larger TD errors are given higher priority and are thus extracted more often during playback.

[0082] During training, the agent does not draw samples uniformly from the experience pool, but rather performs weighted sampling based on priority. Higher-priority experience samples are sampled more frequently, while lower-priority samples are sampled less often. The specific sampling process is based on an importance sampling method, and is performed using the following formula:

[0083] in: For the first Priority of each empirical sample; This is a hyperparameter that controls the degree of influence of priority. When, it is equivalent to uniform sampling; when At that time, samples with higher priority are more likely to be sampled.

[0084] After each experience sample is drawn from the experience pool, the algorithm trains the Q-network based on these samples. The agent's Q-value function is updated using TD error and priority.

[0085] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that the difference between the agent's Q-value function and the Bellman expectation value is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0086] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0087] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the change in the distribution of action choices during the iteration process is less than the policy convergence threshold.

[0088] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0089] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0090] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0091] in: The value of the global reward function. To set a threshold.

[0092] Example 6 Please refer to the example below. Figure 9 As shown, this embodiment further optimizes the previous embodiment by providing a multi-agent reinforcement learning model that simultaneously introduces a global reward decomposition mechanism and a priority experience replay strategy. Based on this model, this embodiment explains the principle of highway hard shoulder management using a high-density interchange group on a highway.

[0093] in, Figure 9 middle Let be the observation sequence of the nth agent.

[0094] Traffic Condition Perception and Modeling: High-density interchange clusters refer to high-volume highway sections with an average spacing of less than 3km and more than 3 consecutive interchanges, typically found in urban agglomerations. This invention targets high-density interchange cluster scenarios on highways, treating interchanges as nodes and the road segments between two nodes as control units. Various traffic condition data can be collected in real time by roadside sensing devices (such as millimeter-wave radar, lidar, and integrated radar-visual systems), including indicators such as traffic flow, speed, and interchange entrance / exit traffic. Each road segment... The state in the control cycle Internally, it can be described as:

[0095] in: For road section Traffic flow in each lane; For road section Upstream interchange Traffic flow at the entrance; For road section Downstream interchange Traffic flow at exits; For road section Average vehicle speed in each lane; The hard shoulder of the current road section is open; The hard shoulder status is from the previous control cycle; Traffic flow ratio between main road and hard shoulder Let M be the dimension of the state space, and M be the space of real numbers.

[0096] Markov Decision Model Construction: In high-density interchange clusters on highways, the dynamic opening decision of hard shoulders is a complex multi-factor optimization problem. This invention models it as a Markov decision model, defining a reward that integrates traffic efficiency, traffic safety, and spatiotemporal consistency to achieve the optimal decision strategy. Based on this model, a multi-agent deep reinforcement learning method is used for training to collaboratively optimize the hard shoulder management method, improving overall traffic efficiency and safety. This includes the following sub-steps: A. State Compression: High-density interchange groups include Each section of road ( , , … Each sub-segment can have its hard shoulder opened / closed independently. Traffic conditions for each segment are as follows: Each traffic characteristic dimension describes the traffic flow (speed, density, traffic volume, openness, etc.). Define the first... road section At any moment The status is:

[0097] The state matrix for the entire road segment is as follows:

[0098] Convert it to a row vector to obtain the original state vector:

[0099] Since directly using high-dimensional states can lead to the curse of dimensionality and excessive computational complexity, Principal Component Analysis (PCA) is used to reduce the dimensionality of the state space. PCA reduces the dimensionality of the state space by analyzing the covariance matrix of historical traffic state data. Perform eigenvalue decomposition to extract the eigenvectors along the direction of maximum variance in the data, and construct a mapping matrix:

[0100] in: It is the eigenvector corresponding to the largest eigenvalue. This represents the dimension of the state space after dimensionality reduction.

[0101] Through the mapping matrix high-dimensional state Projected onto the low-dimensional state vector:

[0102] A low-dimensional state space is defined as:

[0103] B. Action Space Definition: The control strategy is manifested in whether the hard shoulder is enabled on each road segment. Therefore, each road segment corresponds to a binary action variable, indicating whether the hard shoulder is enabled on that segment at the current moment, thus defining the action space. The set of hard shoulder controls for each road segment:

[0104] in: Indicates the first The hard shoulder of this section of road is open at that moment; This indicates that the hard shoulder is open at that moment.

[0105] C. State transition modeling Traffic flow in high-density interchange sections of highways is highly dynamic and uncertain. Its state evolution is affected by multiple factors such as traffic volume, vehicle speed distribution, traffic interference at entrances and exits, and unforeseen events, making the state transition function... It is difficult to accurately describe this using analytical models. Therefore, this invention employs an empirical modeling method based on measured multi-source traffic data. It utilizes traffic operation data collected by roadside sensing devices (such as millimeter-wave radar, lidar, and integrated radar-visual systems) and traffic monitoring systems to construct an approximate state transition function. The specific steps are as follows: ① Data collection: Relying on the roadside sensing and traffic monitoring system, real-time collection of multi-dimensional traffic parameters such as road segment flow, average vehicle speed, density, and entrance / exit flow; ② Input control action: at time Record traffic conditions and the corresponding control actions of the control strategy. (Hard shoulder opening / closing status), and continuously monitor its impact on traffic flow; ③ State transition observation: during action Under its influence, the traffic status at the next moment is observed and recorded. Forming a state-action-result triple. ; ④ Constructing an empirical dataset: By continuously collecting traffic operation data across multiple time periods and scenarios, a large number of state transition samples are accumulated, and then the state transition distribution is approximated and modeled within a reinforcement learning framework.

[0106] This invention, based on an embodiment, uses a reinforcement learning framework to model the dynamic management process of hard shoulders as a multi-agent problem, where each control unit is considered an agent. Each agent aims to maximize the cumulative reward of its control strategy. The state space and action space correspond to different traffic states and hard shoulder control strategies, respectively, and the immediate benefits of actions in a specific state are evaluated using the reward function described in Embodiment 2.

[0107] The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

[0108] Preferably, the training process of the multi-agent deep reinforcement learning model includes: Initialization: Initialize the value functions of all agents. and strategy The initial strategy can be set by random distribution or based on historical traffic statistics; then the agent is iteratively trained using interactive samples driven by multi-source measured traffic data, so that the strategy is gradually optimized under the constraints of real traffic environment characteristics. We collected a large number of state-action-outcome triples generated by Markov decision models from observational data. This data is then added to the experience pool buffer for subsequent training. Action selection and evaluation iteration: for each state The algorithm iterates through all possible actions. Estimate the current action Value, and according to The action that maximizes the value is selected as the current optimal action. Update strategy: The algorithm updates all states based on the result of each iteration. , Includes all states and the corresponding optimal action And push the backtracking time forward; wherein, backtracking is performed at the end of each iteration cycle, starting from the end of the cycle ( Starting from this point, the optimal strategy is gradually pushed back forward, and the agent's strategy is updated accordingly; Termination condition: When the backtracking time is zero (i.e., the start point of the control cycle is reached), the algorithm terminates and returns the globally optimal policy. .

[0109] As a preferred solution, this embodiment introduces a global reward decomposition mechanism and a priority experience replay strategy; The global reward decomposition mechanism is used to decompose global rewards. Decomposed into basic rewards and bonus By allocating different weights, the balance between local and global objectives can be optimized; The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process. The specific formula is as follows:

[0110]

[0111]

[0112] in, , where is the weight learned through the hybrid network, and is the basic reward. Rewards are distributed among the agents primarily through a hybrid network. The basic reward for each agent is... Allocation is based on their contribution to local traffic optimization. Specifically, for intelligent agents... The basic reward it receives can be expressed as:

[0113] in, These are the weights learned through the network.

[0114] Bonus This is used to incentivize agents that contribute significantly to the global optimization process. The gain reward for each agent... The agent's reward is allocated based on its contribution to overall traffic flow optimization. The agent's gain reward is proportional to its contribution, as shown in the following formula:

[0115] in: For intelligent agents Contribution to the overall goal, The total contribution of all intelligent agents.

[0116] In each training iteration, the agent is based on the total reward received. Update its strategy The total reward guides the agent to optimize its decisions, enabling it to strike a balance between local control and global optimization, thereby improving the control effect of the hard shoulder.

[0117] The priority experience replay strategy measures the importance of samples by calculating temporal difference (TD) error and assigns them priority, thereby increasing the probability of key experiences being selected during the sampling process. This method can significantly improve the learning efficiency and convergence speed of the agent.

[0118] In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

[0119] As one possible implementation, multiple agents share an experience pool to fully utilize data generated from interactions between different agents and the environment, promoting collaborative learning. Agents evaluate the importance of each experience using TD error and prioritize experiences based on the absolute value of this error. Specifically, for a state transition triple... The formula for calculating the TD error is as follows:

[0120] Based on the TD error, for each empirical sample Assign a priority:

[0121] in, This is a small constant used to prevent the priority from being zero. In this way, experiences with larger TD errors are given higher priority and are thus extracted more often during playback.

[0122] During training, the agent does not draw samples uniformly from the experience pool, but rather performs weighted sampling based on priority. Higher-priority experience samples are sampled more frequently, while lower-priority samples are sampled less often. The specific sampling process is based on an importance sampling method, and is performed using the following formula:

[0123] in: For the first Priority of each empirical sample; This is a hyperparameter that controls the degree of influence of priority. When, it is equivalent to uniform sampling; when At that time, samples with higher priority are more likely to be sampled.

[0124] After each experience sample is drawn from the experience pool, the algorithm trains the Q-network based on these samples. The agent's Q-value function is updated using TD error and priority.

[0125] The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that the difference between the agent's Q-value function and the Bellman expectation value is less than the convergence threshold of the value function. As a preferred approach, a discount factor is introduced when determining the convergence of the value function. Q-value function of the agent The iteration process should gradually approach the Bellman expectation value. :

[0126] in, This is the current state. For the current action, For the next action, For the next state, For instant rewards, The Q value for the next state; that is This indicates that the Q-function tends to converge; In particular, if the following conditions are met in consecutive iterations k:

[0127] in, The preset convergence threshold can also indicate that the Q function tends to converge; The policy convergence criterion is that the change in the distribution of action choices during the iteration process is less than the policy convergence threshold.

[0128] Policy convergence determination: policy During the iteration process, if the change in the action selection distribution across multiple consecutive iterations is sufficiently small, the policy can be considered convergent.

[0129] in, The policy convergence threshold is typically set to a small positive number (e.g., ...). ).

[0130] In particular, as another possible implementation, convergence can also be determined by combining the global reward fluctuation amplitude during actual training:

[0131] in: The value of the global reward function. To set a threshold.

[0132] Example 7 This embodiment provides a highway hard shoulder management system for traffic resilience, used to execute the method described in any one of embodiments 1-6, including: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models. They are used to generate state-action-result triplets based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by control actions, to represent the decision-making process for the dynamic opening of hard shoulders.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing the hard shoulder of highways based on traffic resilience, characterized in that: include: Collect multi-dimensional traffic data for highways; Input multi-dimensional traffic data from the highway into the trained multi-agent deep reinforcement learning model, and output the decision on whether each hard shoulder of the highway is open at the current moment; The multi-dimensional traffic data of the highway includes traffic flow, vehicle speed and interchange entrance and exit flow. The multi-agent deep reinforcement learning model is trained based on the Markov decision model and uses the reward function to coordinate and optimize the road segments in the interchange group. The reward function includes traffic efficiency reward, traffic safety reward and spatiotemporal consistency reward, and the Markov decision model represents the decision-making process for the dynamic opening of the hard shoulder.

2. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The traffic efficiency reward incorporates total travel time to optimize road network operation efficiency; The total travel time is determined by the inflow of traffic flow, the total outflow of traffic flow, and the time step.

3. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The traffic safety reward is determined based on traffic conflict risk, speed fluctuations, and the frequency of lane-changing behavior. The risk of traffic conflict is determined by the number of vehicles in the road segment; The speed fluctuation is determined by the number of vehicles and their speed within the road segment; The frequency of lane-changing behavior is determined by the number of times a vehicle changes lanes.

4. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The spatiotemporal consistency reward includes a temporal coordination reward and a spatial coordination reward, which are used to control the frequency of hard shoulder opening and closing and the consistency of opening and closing states between adjacent road segments, respectively.

5. The highway hard shoulder management method for traffic resilience according to claim 1, characterized in that, The multi-agent deep reinforcement learning model includes several agents, each corresponding to an interchange segment. The control unit of each agent collects multi-dimensional traffic data for its respective segment and outputs a decision on whether to open the hard shoulder corresponding to the segment.

6. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The training process of the multi-agent deep reinforcement learning model includes: The experience pool is filled with state-action-outcome triples generated by the Markov decision model, and the value function and policy of all agents are initialized. For each state, iterate through all possible actions and calculate the Q value corresponding to the action. Select the action with the maximum value as the current optimal action. Starting from the end of the cycle, we trace back time and take all states and their corresponding optimal actions as the current optimal strategy. When the backtracking time is zero, output the current optimal strategy.

7. The highway hard shoulder management method for traffic resilience according to claim 6, characterized in that, The convergence criteria for the multi-agent deep reinforcement learning model include value function convergence and policy convergence. The convergence criterion for the value function is that it converges when the difference between the agent's Q-value function and the Bellman expectation is less than the convergence threshold of the value function. The policy convergence criterion is that the agent's policy converges when the magnitude of the change in the action selection distribution during the iteration process is less than the policy convergence threshold.

8. The highway hard shoulder management method for traffic resilience according to claim 1, characterized in that, The multi-agent deep reinforcement learning model includes a global reward decomposition mechanism to decompose the global reward into a basic reward and a gain reward. The basic reward is used to allocate learning weights to each agent, while the gain reward is allocated based on the agent's contribution to the overall traffic flow optimization and is used to incentivize the agent during the global optimization process.

9. The highway hard shoulder management method based on traffic resilience according to claim 1, characterized in that, The multi-agent deep reinforcement learning model also includes a priority experience replay strategy; In the priority experience replay strategy, multiple agents share an experience pool. The importance of each experience sample in the experience pool is evaluated by the time difference error, and the experience sample is assigned a priority based on the absolute value of the time difference error. The agent performs weighted sampling of experience samples according to their priority and then trains a Q-network.

10. A highway hard shoulder management system for traffic resilience, characterized in that: For performing the method according to any one of claims 1-9, comprising: Traffic data acquisition unit, used to collect multi-dimensional traffic data of highways; A multi-agent deep reinforcement learning model is used to output the opening strategy of each hard shoulder of the highway at the current moment based on multi-dimensional traffic data of the highway. Markov decision models participate in the training process of multi-agent deep reinforcement learning models. They are used to generate state-action-result triplets based on multi-dimensional traffic data of highways, control actions corresponding to management strategies, and the states generated by control actions, to represent the decision-making process for the dynamic opening of hard shoulders.

Images