A multi-strategy cooperative optimization method for scalable expressway traffic flow management

Abstract

This invention discloses a scalable multi-strategy collaborative optimization method for highway traffic flow control, comprising: S1. Defining corresponding reinforcement learning action spaces based on different types of traffic flow control strategies; S2. Defining corresponding reinforcement learning state spaces based on traffic flow observation states; S3. Defining reward functions for reinforcement learning based on different target coefficients; S4. Constructing a simulation training environment; S5. Constructing reinforcement learning agent strategies based on action spaces, state spaces, reward functions, and the simulation training environment; S6. Optimizing the constructed reinforcement learning agent strategies in the simulation training environment to obtain the optimal reinforcement learning agent strategy; S7. Obtaining the actual environmental state and calculating the optimal traffic flow control strategy for the actual environmental state based on the optimal reinforcement learning agent strategy.

Description

Technical Field

[0001] This invention relates to the field of highway traffic management technology, and in particular to a scalable multi-strategy collaborative optimization method for highway traffic flow management. Background Technology

[0002] With the development of intelligent transportation, its application in highway traffic flow management is receiving increasing attention. Highway sensing equipment collects traffic and weather data through various sensors, processes the collected data using information technology equipment and software platforms, and ultimately presents it in different ways at the application layer to improve traffic service quality. As a crucial component of the application layer, the traffic flow management system makes decisions on management strategies based on traffic flow conditions and disseminates this information through information dissemination devices, effectively regulating traffic flow. With the development and application of traffic flow management technology, traffic flow management strategies are constantly being enriched. How to effectively coordinate various management strategies to further improve highway traffic safety and efficiency is of great significance.

[0003] Existing technologies also include some research on traffic flow control. For example, patent application number 2022114724056 discloses a highway traffic control method based on reinforcement learning, which mainly utilizes reinforcement learning data-driven methods to optimize a single traffic control strategy. Patent application number 2024105407291 discloses a highway ramp control method based on reinforcement learning and event-triggered prediction, which mainly utilizes reinforcement learning and model predictive control to optimize a single ramp control strategy. While these existing technologies have achieved certain results, their content only addresses a single traffic control strategy and differs from the practical application of multiple control strategies working together.

[0004] To address the aforementioned technical problems, this invention proposes a scalable multi-strategy collaborative optimization method for highway traffic flow management. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a scalable multi-strategy collaborative optimization method for highway traffic flow management.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A scalable multi-strategy collaborative optimization method for highway traffic flow management includes:

[0008] S1. Define the corresponding reinforcement learning action space based on different types of traffic flow control strategies;

[0009] S2. Reinforcement learning state space based on traffic flow observation state definition;

[0010] S3. Define the reward function for reinforcement learning based on different target coefficients;

[0011] S4. Construct a simulation training environment;

[0012] S5. Construct reinforcement learning agent strategies based on action space, state space, reward function, and simulation training environment;

[0013] S6. Optimize the constructed reinforcement learning agent policy in a simulation training environment to obtain the optimal reinforcement learning agent policy;

[0014] S7. Obtain the actual environmental state and calculate the optimal traffic flow control strategy based on the optimal reinforcement learning agent strategy.

[0015] Furthermore, step S1 includes:

[0016] S11. Obtain the parameters corresponding to different types of control strategies;

[0017] S12. Obtain the types of traffic flow control strategies that can be implemented on the road segment;

[0018] S13. Define the reinforcement learning action space based on the types of traffic flow control strategies that can be implemented on the road segment and the corresponding parameters.

[0019] Furthermore, the reinforcement learning action space in step S13 is represented as follows:

[0020]

[0021] in, Represents the action space; a 限速 Indicates parameters for dynamic speed limiting control strategy; a 硬路肩 Indicates the parameters of the hard shoulder opening control strategy; a 动态车道 Indicates parameters for dynamic lane management and control strategies; a 匝道 Indicates the parameters of the ramp flow regulation and control strategy; a 动态路径 Indicates parameters for dynamic path guidance and control strategies; a 其他 This indicates other control strategy parameters.

[0022] Furthermore, step S2 includes:

[0023] S21. Obtain the state parameters corresponding to the traffic flow observation state;

[0024] S22. Obtain the status observation points of the road segment;

[0025] S23. Define the reinforcement learning state space based on the state observation points and corresponding state parameters of the road segment.

[0026] Furthermore, the reinforcement learning state space in step S23 is represented as follows:

[0027]

[0028] in, Represents the state space; s1 represents the state parameters of the first observation point; s n s represents the state parameters of the nth observation point; N This represents the state parameters of the Nth observation point.

[0029] Furthermore, the different target coefficients in step S3 include the total travel time of vehicles on the road segment and the ramp queue length penalty factor;

[0030] The total travel time for vehicles on this route is expressed as follows:

[0031]

[0032] Among them, L T T represents the total travel time of vehicles on a road segment; i represents a specific road segment; I represents the total travel time of vehicles on a road segment. all L represents the set of all road segments; i λ represents the length of road segment i; i ρ represents the number of lanes in road segment i; i (k) represents the flow density of road segment i in time period k; I on This represents the set of road segments connecting to the entrance ramp; w i (k) represents the queue length of the entrance ramp of road segment i in time period k;

[0033] The ramp queue length penalty factor is expressed as:

[0034]

[0035] Among them, L W σ represents the queue length penalty factor; i (k) represents the queue length penalty factor for segment i in time period k;

[0036] The reward function for reinforcement learning in step S3 is expressed as:

[0037]

[0038] in, L represents the reward function; O These represent the coefficients of other objective functions.

[0039] Furthermore, the construction of the simulation training environment in step S4 is represented as follows:

[0040]

[0041] in, f represents the state space for the next time interval k+1; tra The computational function representing the state space; Represents the state space of the current time period k; This represents the action space of the current time period k; G represents the reward function for the current time period k; tra This represents the function used to calculate the reward function.

[0042] Furthermore, step S5 includes:

[0043] S51. Initialize the reinforcement learning algorithm policy parameters and initialize the number of training iterations;

[0044] S52. Initialize the state space and reward function, and initialize the number of iterations;

[0045] S53. Obtain the action space for the current time period based on parameters of traffic flow control strategy type;

[0046] S54. Calculate the state space for the next time period and the reward function for the current time period based on the simulation training environment;

[0047] S55. Update the algorithm policy parameters of the reinforcement learning algorithm based on the action space of the current time period, the state space of the current time period, the state space of the next time period, and the reward function of the current time period;

[0048] S56. Determine if the number of iterations is less than the total number of iterations. If yes, proceed to step S53; otherwise, proceed to step S57.

[0049] S57. Determine if the number of training iterations is less than the total number of training iterations. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy.

[0050] Furthermore, in step S53, the action space for the current time period is obtained based on the parameters of the traffic flow control strategy type, as expressed as:

[0051]

[0052] in, Represents the action space of the current time period k; π θ This represents the reinforcement learning agent's strategy; Represents the state space of the current time period k;

[0053] In step S55, the algorithm policy parameters for updating the reinforcement learning algorithm policy based on the action space of the current time period, the state space of the current time period, the state space of the next time period, and the reward function of the current time period are expressed as follows:

[0054]

[0055] Where θ represents the policy parameters of the reinforcement learning agent algorithm; Represents the state space for the next time interval k+1; Let Q represent the reward function for the current time period k; let Q represent the policy parameter update function of the reinforcement learning agent algorithm.

[0056] Furthermore, the optimal traffic flow control strategy for calculating the actual environmental conditions in step S7 is expressed as:

[0057] a * =π * (s real )

[0058] Where a* represents the optimal traffic flow control strategy; π* represents the optimal reinforcement learning agent strategy; s real This indicates the actual environmental conditions.

[0059] Compared with existing technologies, this invention proposes a multi-strategy collaborative optimization method for highway traffic flow management. It utilizes a reinforcement learning agent to make intelligent decisions and determine strategy types and parameters. During this process, a simulation environment is constructed to continuously train the agent, and a reasonable reward and action space are designed to improve the decision-making performance of the reinforcement learning agent. The trained reinforcement learning agent can be deployed into the traffic flow management system, effectively improving the intelligent highway traffic flow management effect. Thanks to the reasonable construction of the action and state spaces of the reinforcement learning agent, this invention is applicable to various traffic flow management scenarios. It only requires adding the strategies and available states of the management scenario to the corresponding action and state spaces and training the agent to obtain the collaborative optimization algorithm. No adjustments to the optimization algorithm architecture and content functions are needed, resulting in strong overall scalability. Especially after the update of road segment management equipment and the addition of a large number of sensing and information dissemination devices, the technical method provided by this invention can expand to new actions and states without changing the overall architecture. Attached Figure Description

[0060] Figure 1 This is a flowchart of a scalable multi-strategy collaborative optimization method for highway traffic flow management provided in Implementation Example 1;

[0061] Figure 2 This is a schematic diagram of the deployment of multi-control strategy equipment on a highway section provided in Example 2;

[0062] Figure 3 This is a schematic diagram of the deployment of multi-control strategy equipment on a highway section provided in Example 3;

[0063] Figure 4This is a schematic diagram of the deployment of multi-control strategy equipment on a highway section provided in Example 4. Detailed Implementation

[0064] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0065] The purpose of this invention is to address the shortcomings of existing technologies by providing a scalable multi-strategy collaborative optimization method for highway traffic flow management.

[0066] Example 1

[0067] This embodiment provides a scalable multi-strategy collaborative optimization method for highway traffic flow management, such as... Figure 1 As shown, it includes:

[0068] S1. Define the corresponding reinforcement learning action space based on different types of traffic flow control strategies;

[0069] S2. Reinforcement learning state space based on traffic flow observation state definition;

[0070] S3. Define the reward function for reinforcement learning based on different target coefficients;

[0071] S4. Construct a simulation training environment;

[0072] S5. Construct reinforcement learning agent strategies based on action space, state space, reward function, and simulation training environment;

[0073] S6. Optimize the constructed reinforcement learning agent policy in a simulation training environment to obtain the optimal reinforcement learning agent policy;

[0074] S7. Obtain the actual environmental state and calculate the optimal traffic flow control strategy based on the optimal reinforcement learning agent strategy.

[0075] In step S1, a corresponding reinforcement learning action space is defined based on different types of traffic flow control strategies; specifically including:

[0076] S11. Obtain the parameters corresponding to different types of control strategies; among which, control strategies include dynamic speed limit, hard shoulder opening, dynamic lane management, ramp flow regulation, dynamic path guidance, and other control strategies; the action space parameters corresponding to the control strategy are shown in Table 1 below:

[0077] Table 1. Parameters of the action space involved in different types of control strategies

[0078] Control strategy types motion space parameters Dynamic speed limiting Speed ​​limit, control area (start and end locations), control period (start and end times) Hard shoulder open Open / close instructions, control scope (start and end locations), control period (start and end times). Dynamic lane management Lane opening / closing instructions, controlled area (start and end locations), controlled period (start and end times). Ramp flow regulation Regulation rate, location of controlled ramps, and control period (start and end times). Dynamic path guidance Route selection, control and guidance locations, control periods (start and end times) Other control strategies Key parameters of the control strategy, the period of application of the control strategy, and the location of application of the control strategy.

[0079] S12. Obtain the types of traffic flow control strategies that can be implemented in the road segment;

[0080] S13. Define the reinforcement learning action space based on the types of traffic flow control strategies that can be implemented on the road segment and the corresponding parameters; whereby the reinforcement learning action space is represented as:

[0081]

[0082] in, Represents the action space; a 限速 Indicates parameters for dynamic speed limit control strategy; a 硬路肩 Indicates the parameters of the hard shoulder opening control strategy; a 动态车道 Indicates parameters for dynamic lane management and control strategies; a 匝道 Indicates the parameters of the ramp flow regulation and control strategy; a 动态路径 Indicates the parameters of the dynamic path guidance and control strategy; a 其他 This indicates other control strategy parameters.

[0083] In step S2, a corresponding reinforcement learning state space is defined based on the traffic flow observation state; specifically, this includes:

[0084] S21. Obtain the state space parameters corresponding to the traffic flow observation state; the state space parameters mainly include cross-sectional average speed, cross-sectional average flow, road segment vehicle density, road segment queue length, weather parameters, and lane occupancy location.

[0085] S22. Obtain the state observation points in the road segment, and the state space parameters that can be obtained for each observation point;

[0086] S23. Define the reinforcement learning state space based on the state observation points in the road segment and the state parameters obtainable at each observation point; wherein, the reinforcement learning state space is represented as:

[0087]

[0088] in, Represents the state space; s1 represents the state parameters of the first observation point; s n s represents the state parameters of the nth observation point; N This represents the state parameters of the Nth observation point.

[0089] In step S3, the reward function for reinforcement learning is defined based on different target coefficients; specifically, it includes:

[0090] S31. Define the calculation methods for different objective coefficients; among which, different objective coefficients include total travel time of vehicles on road segments and ramp queue length penalty factors; objective coefficients may sometimes also include other objective function coefficients;

[0091] The total travel time for vehicles on this route is expressed as follows:

[0092]

[0093] Among them, L T T represents the total travel time of vehicles on a road segment; i represents a specific road segment; I represents the total travel time of vehicles on a road segment. all L represents the set of all road segments; i λ represents the length of road segment i; i ρ represents the number of lanes in road segment i; i (k) represents the flow density of road segment i in time period k; I on This represents the set of road segments connecting to the entrance ramp; w i (k) represents the queue length of the entrance ramp of road segment i in time period k;

[0094] The ramp queue length penalty factor is expressed as:

[0095]

[0096] Among them, L W σ represents the queue length penalty factor; i (k) represents the queue length penalty factor for segment i in time period k;

[0097] S32. Define the reinforcement learning reward function according to different target coefficients, expressed as:

[0098]

[0099] in, L represents the reward function; O These represent the coefficients of other objective functions.

[0100] This embodiment uses a reinforcement learning reward function to link the control strategy in the action space with the traffic flow observation state in the state space, providing incentive signals for the agent's decision-making. The reward function not only evaluates the agent's behavior based on action space parameters but also adjusts the reward value according to changes in state space parameters, thereby guiding the agent to select the optimal control strategy in a dynamic traffic environment to achieve the optimization goal.

[0101] In step S4, a simulation training environment is constructed.

[0102] Simulation training environments are fundamental to training reinforcement learning agents. They simulate the real-world operation of highway traffic flow, providing the agent with an interaction scenario similar to the actual environment. Through simulation environments, the agent can try different control strategies in a virtual setting and learn and optimize based on feedback from the reward function.

[0103] Methods for constructing simulation training environments include:

[0104] S41. Divide the simulated road sections according to the status observation points and the scope of the control strategy implementation;

[0105] State observation points: Based on the state space parameters defined in step S2 (such as cross-sectional average speed, cross-sectional average flow rate, road segment vehicle density, etc.), determine the locations of observation points on the simulated road segment. These observation points should cover key areas, such as entrance ramps, exit ramps, main roads, hard shoulders, etc.

[0106] Scope of Control Strategy Implementation: Based on the control strategy type defined in step S1 (such as dynamic speed limit, hard shoulder opening, ramp flow regulation, etc.), determine the scope of implementation for each strategy. For example, a dynamic speed limit strategy may be applied to a specific section of a main road, while a ramp flow regulation strategy may be applied to entrance ramps.

[0107] The simulation road segment is divided into multiple sub-regions, each corresponding to one or more observation points and the scope of implementation of control strategies. This division method facilitates the simulation of the impact of different strategies on traffic flow in the simulation environment.

[0108] S42. Construct a simulation training environment based on traffic flow mathematical models or simulation software, taking into account state space and action space.

[0109] S421. Select a suitable traffic flow model or simulation software.

[0110] Traffic flow mathematical models are based on classical traffic flow theories (such as the Lighthill-Whitham-Richards model and car-following model) and construct mathematical models to describe the dynamic characteristics of traffic flow. These models can be used to calculate parameters such as speed, volume, and density of traffic flow.

[0111] The simulation software used is professional traffic simulation software (such as VISSIM, AIMSUN, Synchro, etc.). These software programs provide rich traffic flow modeling functions and can simulate real traffic scenarios, including vehicle driving, traffic signal control, ramp flow regulation, etc.

[0112] S422. Construct a simulation training environment, which includes road segment modeling, traffic flow generation, observation point setting, and control strategy implementation.

[0113] Road segment modeling: Simulated road segments are constructed based on the geometric characteristics of actual highways (such as the number of lanes, length, and gradient) and traffic facilities (such as ramps and hard shoulders).

[0114] Traffic flow generation: Generates traffic flow based on actual traffic volume data or prediction models. Different traffic flow scenarios can be set, such as traffic flow during peak hours, off-peak hours, and inclement weather conditions.

[0115] Observation point setup: Set up virtual sensors on the simulated road segment that correspond to the state space observation points defined in step two to acquire traffic flow state parameters (such as speed, flow rate, density, etc.) in real time.

[0116] Implementation of control strategies: Integrate the control strategies defined in step S1 (such as dynamic speed limits, hard shoulder opening, etc.) into the simulation environment. Dynamic adjustments to these strategies are achieved through programming interfaces or the control functions of the simulation software.

[0117] In this embodiment, the constructed simulation training environment is represented as follows:

[0118]

[0119] in, f represents the state space for the next time interval k+1; tra The computational function representing the state space; Represents the state space of the current time period k; This represents the action space of the current time period k; G represents the reward function for the current time period k; tra This represents the function used to calculate the reward function.

[0120] In this embodiment, constructing a simulation training environment is a crucial step in training a reinforcement learning agent. By rationally dividing the simulated road segments, selecting appropriate traffic flow models or simulation software, and integrating the state space and action space into the simulation environment, a training platform similar to the actual traffic environment can be provided for the agent. The simulation environment can not only simulate the dynamic characteristics of traffic flow but also provide the agent with real-time state feedback and reward signals, thereby enabling the optimization and learning of control strategies.

[0121] In step S5, a reinforcement learning agent policy is constructed based on the action space, state space, reward function, and simulation training environment; specifically, this includes:

[0122] S51. Initialize the policy parameters θ of the selected reinforcement learning algorithm, initialize the number of training iterations e = 0, and set the total number of training iterations E;

[0123] Reinforcement learning algorithms can include Q-learning, Deep Q-Networks [DQN], Policy Gradient, etc.; the policy parameter θ determines the probability or direct mapping of the agent's choice of action in a given state; the number of training iterations determines the total number of training rounds for the agent, and the number of iterations determines the number of steps the agent interacts with the environment in each training round.

[0124] S52. Initialize the state space and reward function, initialize the number of iterations k = 0, and set the total number of iterations K;

[0125] Based on the state space parameters defined in step S2 (such as cross-sectional average speed, flow rate, vehicle density, etc.), the initial state space is obtained from the simulation environment.

[0126] Initialize the reward function value according to the reward function defined in step S3 (such as reducing the total travel time of vehicles, reducing the queue length of ramps, etc.).

[0127] S53. Obtain the action space for the current time period based on the parameters of the traffic flow control strategy type.

[0128] Based on the current state and policy parameter π θ The agent selects an action. The method of selecting an action depends on the reinforcement learning algorithm used, and it must be ensured that the selected action is within the action space defined in step S1 (such as dynamic speed limit value, ramp flow regulation rate, etc.), expressed as:

[0129]

[0130] in, Represents the action space of the current time period k; π θ This represents the reinforcement learning agent's strategy; Represents the state space of the current time period k;

[0131] S54. Calculate the state space for the next time period based on the simulation training environment. and the reward function for the current time period

[0132] Select the action It is applied to simulation training environments, which calculate the state of the next stage based on traffic flow models or simulation software.

[0133] According to the reward function defined in step S3 Calculate the reward value of the current action. The reward value reflects the action. Contribution to optimization goals (such as reducing travel time and alleviating congestion).

[0134] S55. Action space based on the current time period State space of the current time period The state space of the next time period Reward function for the current time period Update the algorithm policy. The reinforcement learning algorithm policy parameters are represented as follows:

[0135]

[0136] Where θ represents the policy parameters of the reinforcement learning agent algorithm; Represents the state space for the next time interval k+1; Let Q represent the reward function for the current time period k; let Q represent the policy parameter update function of the reinforcement learning agent algorithm.

[0137] S56. Determine if the number of iterations is less than the total number of iterations. If yes, proceed to step S53; otherwise, proceed to step S57.

[0138] In this embodiment, the strategy parameters are updated based on feedback to optimize the behavior of the agent;

[0139] S57. Determine if the number of training iterations is less than the total number of training iterations. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy.

[0140] In this embodiment, the optimal strategy is gradually approximated through multiple training and iterations.

[0141] Based on the above operations, the intelligent agent can learn how to select the optimal combination of control strategies in complex traffic flow environments, thereby achieving optimized management of highway traffic flow.

[0142] In step S6, the constructed reinforcement learning agent policy is optimized in a simulation training environment to obtain the optimal reinforcement learning agent policy.

[0143] This embodiment applies the reinforcement learning algorithm designed in step S5 to the simulation training environment of step S4, and through training, enables the agent to learn the optimal traffic flow control strategy; specifically including:

[0144] S61. Deploy the simulation training environment;

[0145] Deploy the simulation training environment built in step S4 into the training system. Ensure that the simulation environment can provide real-time feedback on traffic flow status information (such as speed, flow rate, vehicle density, etc.) and support the execution of actions by the intelligent agent (such as dynamic speed limits, ramp flow regulation, etc.).

[0146] Adjust the parameters of the simulation environment according to actual needs, such as traffic flow generation rate, road segment length, and observation point location, to simulate different traffic scenarios (such as peak hours, severe weather, etc.).

[0147] Ensure that the reinforcement learning algorithm can seamlessly interface with the simulation environment to acquire state information and execute actions.

[0148] S62. Select different types of reinforcement learning agent algorithm policy parameter update functions Q for training;

[0149] Based on the complexity of the problem and the actual needs, select a suitable reinforcement learning algorithm, and initialize the relevant parameters according to the selected algorithm, for example:

[0150] Learning rate (α): The rate at which control parameters are updated.

[0151] Discount factor (γ): determines the weight of future rewards.

[0152] Exploration rate (ε): The balance between exploration and exploitation of the control agent.

[0153] Neural network weights: If using DQN or Policy Gradient, initialize the weights of the neural network.

[0154] Other hyperparameters include the target network's update frequency (DQN) and batch size (used for training neural networks).

[0155] S63. After training, select the reinforcement learning agent policy π* that has the best performance.

[0156] Based on the algorithm logic designed in step S5, multiple rounds of training are performed to obtain the reinforcement learning agent policy.

[0157] This embodiment also includes recording key indicators during the training process, such as changes in reward value, policy convergence, and training time; dynamically adjusting hyperparameters such as learning rate and exploration rate based on the performance during training to improve training effectiveness; and periodically saving policy parameters during training so that training can be resumed if interrupted.

[0158] Based on the performance evaluation results during training, select the reinforcement learning strategy that performs best. The following criteria can be considered:

[0159] Reward maximization: Choose the strategy with the highest average reward value.

[0160] Traffic flow optimization: Select the strategy that has the best effect on optimizing traffic flow indicators (such as congestion relief and travel time reduction).

[0161] Stability: Select a strategy whose parameters converge stably and are robust.

[0162] Save the optimal strategy: Save the parameters of the optimal strategy for subsequent deployment and practical application.

[0163] Through the above operations, the intelligent agent can learn in a virtual environment how to select the optimal control strategy under complex and ever-changing traffic flow conditions, thereby providing effective decision support for actual highway traffic flow control.

[0164] In step S7, the actual environmental state is obtained, and the optimal traffic flow control strategy for the actual environmental state is calculated based on the optimal reinforcement learning agent strategy.

[0165] Deploying reinforcement learning agents is a crucial step in applying trained and validated optimal reinforcement learning agent policies to real-world traffic flow management systems. The goal is to deploy the optimal policies learned by the agent in a simulation environment to a real-world highway information system, enabling it to optimize traffic flow management strategies in real-time. Specifically, this includes:

[0166] S71. Deploy the best-performing reinforcement learning agent policy π* into the local information system;

[0167] Ensure that the actual highway information system has sufficient hardware resources (such as servers, sensor networks, and communication equipment) to support the operation of the intelligent agent. Simultaneously, the software environment needs to be compatible with the simulation training environment, capable of receiving the agent's policy output and executing corresponding control actions.

[0168] Establish data interfaces between the intelligent agent and the actual traffic flow monitoring system. These interfaces need to be able to acquire real-time traffic flow status data (such as vehicle speed, flow rate, vehicle density, etc.) and transmit the intelligent agent's decision-making instructions to traffic control equipment (such as speed limit signs, lane indicators, ramp traffic lights, etc.).

[0169] Deploy intelligent agents into local information systems to ensure seamless integration with existing traffic management systems (such as traffic signal control systems, information dissemination systems, etc.).

[0170] S72. Based on the collected actual environmental conditions, calculate the optimal traffic flow control strategy for the actual environmental conditions, expressed as:

[0171] a * =π * (s real )

[0172] Where a* represents the optimal traffic flow control strategy; π* represents the optimal reinforcement learning agent strategy; s real This indicates the actual environmental conditions.

[0173] In a real-world operating environment, the agent's policy parameters are calibrated. Since real-world environments may differ from simulation environments (e.g., traffic flow, road conditions, driving behavior), short-term test runs are necessary to adjust the policy parameters and ensure they adapt to the real-world environment. Initial state data (e.g., road segment vehicle density, average speed, ramp queue length) is obtained from a real-world traffic flow monitoring system and used as input for the agent's decision-making. Small-scale tests are conducted in a real-world environment to verify the effectiveness of the agent's policy and the system's stability. Manual intervention is possible during testing to ensure the system's safety and reliability.

[0174] Based on the current traffic flow, the agent uses pre-trained strategies to calculate the optimal control policy (such as dynamic speed limits, ramp flow regulation rates, lane opening / closing instructions, etc.). The control policy output by the agent is then transmitted to traffic control equipment (such as variable speed limit signs, ramp traffic lights, lane indicators, etc.) via a communication interface, and these devices execute the corresponding control actions. The agent can dynamically adjust the control policy based on real-time traffic flow changes. For example, when congestion is detected on a road segment, the agent can adjust the speed limit or open the hard shoulder in real time to alleviate congestion.

[0175] This embodiment also includes monitoring and feedback of the intelligent agent, specifically:

[0176] Real-time monitoring of the agent's operational status and traffic flow changes. Key traffic indicators (such as average vehicle speed, queue length, and congestion index) are displayed through a visual interface, allowing operators to understand the system's operational status promptly. The performance of the agent's strategies is periodically evaluated to check if the expected optimization goals (such as reducing total vehicle travel time and lowering ramp queue length) are achieved. If performance degradation or anomalies are detected, strategy parameters are adjusted or system maintenance is performed promptly. A feedback mechanism is established, allowing operators to manually intervene in the agent's decisions based on actual conditions. For example, in the event of special events (such as traffic accidents or severe weather), operators can adjust the agent's strategies or directly take over control of the equipment.

[0177] Based on the above approach, intelligent agents can dynamically optimize traffic flow control strategies in real-world highway environments, thereby improving the operational efficiency and safety of the transportation system.

[0178] Compared with existing technologies, this embodiment proposes a multi-strategy collaborative optimization method for highway traffic flow management. It utilizes a reinforcement learning agent to make intelligent decisions and determine strategy types and parameters. During this process, a simulation environment is constructed to continuously train the agent, and a reasonable reward and action space are designed to improve the decision-making performance of the reinforcement learning agent. The trained reinforcement learning agent can be deployed into the traffic flow management system, effectively improving the intelligent highway traffic flow management effect. Thanks to the reasonable construction of the action and state spaces of the reinforcement learning agent, this invention is applicable to various traffic flow management scenarios. It only requires adding the strategies and available states of the management scenario to the corresponding action and state spaces and training the agent to obtain the collaborative optimization algorithm. No adjustments to the optimization algorithm architecture and content functions are needed, resulting in strong overall scalability. Especially after the update of road segment management equipment and the addition of a large number of sensing and information dissemination devices, the technical method provided by this invention can expand to new actions and states without changing the overall architecture.

[0179] Example 2

[0180] The scalable multi-strategy collaborative optimization method for highway traffic flow management provided in this embodiment differs from Embodiment 1 in that:

[0181] like Figure 2 As shown, the control strategies in this embodiment include dynamic speed limit, hard shoulder opening, and ramp flow regulation. Therefore, a corresponding dynamic speed limit information board, a hard shoulder opening information board, and a ramp flow regulation information board are set up on the road segment.

[0182] S1. Define the reinforcement learning action space based on the control policy type, specifically as follows:

[0183] S11. Define the action space parameters involved in different types of control strategies, as shown in Table 2 below:

[0184] Table 2. Parameters of the action space involved in different types of control strategies

[0185] Control strategy types motion space parameters Dynamic speed limiting Speed ​​limit, control area (start and end locations), control period (start and end times) Hard shoulder open Open / close instructions, control scope (start and end locations), control period (start and end times). Ramp flow regulation Regulation rate, location of controlled ramps, and control period (start and end times).

[0186] S12. Identify the types of traffic flow control strategies that can be implemented on the road segment. In this embodiment, these include dynamic speed limits, hard shoulder opening, and ramp flow regulation.

[0187] S13. Define the action space based on the types of traffic flow control strategies that can be implemented on the road segment and the parameters of different control strategies, as follows:

[0188]

[0189] S2. The reinforcement learning state space is defined based on the traffic flow observation state, specifically as follows:

[0190] S21. Define common state-space parameters, mainly including: cross-sectional average speed, cross-sectional average flow rate, road segment vehicle density, and road segment queue length;

[0191] S22. Organize the status observation points of the road section and the status parameters that can be obtained from each observation point. This embodiment includes a total of three observation points, and the average cross-sectional speed and average cross-sectional flow can be obtained from all three observation points.

[0192] S23. Define the state space based on the state observation points of the road segment and the state parameters obtainable at each observation point, as follows:

[0193]

[0194] S3. Define reinforcement learning rewards based on the optimization objective, specifically as follows:

[0195] S31. Define the calculation methods for different target coefficients;

[0196] a) Formula for calculating the total travel time of vehicles on a road segment:

[0197]

[0198] b) Formula for calculating ramp queue length penalty factor:

[0199]

[0200] S33. In this embodiment, L is defined. O The action space cost is represented as:

[0201] L O =∑a(k)

[0202] The reinforcement learning reward is defined based on various target coefficients and is expressed as follows:

[0203]

[0204] This reward consists of three parts: total vehicle travel time, ramp queue length penalty factor, and motion space cost.

[0205] S4. Construct a simulation training environment, specifically as follows:

[0206] S41. Divide the simulated road segments according to the state space observation points and the scope of the control strategy implementation.

[0207] S42. In this embodiment, simulation software is used to construct a simulation training environment, which is based on state space and action space.

[0208]

[0209] Among them, f tra and g tra It can be obtained through simulation software.

[0210] S5. Reinforcement learning agent algorithm design, specifically:

[0211] S51. Initialize the reinforcement learning algorithm policy parameters θ, initialize the number of training iterations e = 0, and the total number of training iterations E = 1000.

[0212] S52. Initialize the state space and reward function, and initialize the number of iterations k = 0, with a total number of iterations K = 200;

[0213] S53. The action space for the current time period is obtained based on the parameters of the traffic flow control strategy type, represented as:

[0214]

[0215] S54. Calculate the state space for the next time period based on the simulation training environment. and the reward function for the current time period

[0216] S55. Action space based on the current time period State space of the current time period The state space of the next time period Reward function for the current time period Update the algorithm policy. The reinforcement learning algorithm policy parameters are represented as follows:

[0217]

[0218] S56. Determine if the number of iterations is less than the total number of iterations K. If yes, proceed to step S53; otherwise, proceed to step S57.

[0219] S57. Determine if the number of training iterations is less than the total number of training iterations E. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy.

[0220] In step S6, the reinforcement learning agent is trained, specifically as follows:

[0221] S61. Deploy the simulation training environment built in step S4.

[0222] S62. Select different types of reinforcement learning agent algorithm policy parameter update functions Q for training. In this embodiment, Q-learning, DQN, and PPO algorithms are used for training respectively.

[0223] S63. After training, select the reinforcement learning agent policy π* that has the best performance.

[0224] In step S7, the reinforcement learning agent is deployed.

[0225] S71. Deploy the best-performing reinforcement learning agent policy π* into the local information system.

[0226] S72. Based on the collected actual environmental conditions, calculate the optimal traffic flow control strategy for the actual environmental conditions, expressed as:

[0227] a * =π * (s real )

[0228] Among them, a* includes parameters for three strategies: dynamic speed limiting, hard shoulder opening, and ramp flow regulation; s real The state variables include three observation points.

[0229] Example 3

[0230] The scalable multi-strategy collaborative optimization method for highway traffic flow management provided in this embodiment differs from Embodiment 1 in that:

[0231] like Figure 3 As shown, the control strategy type in this embodiment includes dynamic lane management, so a corresponding dynamic lane management information board is set up on the road segment.

[0232] S1. Define the reinforcement learning action space based on the control policy type, specifically as follows:

[0233] S11. Define the action space parameters involved in different types of control strategies, as shown in Table 3 below:

[0234] Table 3. Parameters of the action space involved in different types of control strategies

[0235]

[0236] S12. Identify the types of traffic flow control strategies that can be implemented on the road segment. In this embodiment, these include dynamic speed limits, hard shoulder opening, and ramp flow regulation.

[0237] S13. Define the action space based on the types of traffic flow control strategies that can be implemented on the road segment and the parameters of different control strategies, as follows:

[0238]

[0239] S2. The reinforcement learning state space is defined based on the traffic flow observation state, specifically as follows:

[0240] S21. Define common state-space parameters, mainly including: cross-sectional average speed, cross-sectional average flow rate, road segment vehicle density, and road segment queue length;

[0241] S22. Organize the status observation points of the road segment and the status parameters that can be obtained from each observation point. This embodiment includes two observation points, and both observation points can obtain the cross-sectional average speed and cross-sectional average flow.

[0242] S23. Define the state space based on the state observation points of the road segment and the state parameters obtainable at each observation point, as follows:

[0243]

[0244] S3. Define reinforcement learning rewards based on the optimization objective, specifically as follows:

[0245] S31. Define the calculation methods for different target coefficients;

[0246] a) Formula for calculating the total travel time of vehicles on a road segment:

[0247]

[0248] b) Formula for calculating ramp queue length penalty factor:

[0249]

[0250] S33. L is not defined in this embodiment. O The reinforcement learning reward is defined based on the coefficients of various objectives, and is expressed as follows:

[0251]

[0252] This reward includes two parts: total vehicle travel time and ramp queue length penalty factor.

[0253] S4. Construct a simulation training environment, specifically as follows:

[0254] S41. Divide the simulated road segments according to the state space observation points and the scope of the control strategy implementation.

[0255] S42. Regarding the state space and action space, this embodiment uses a traffic flow mathematical model to construct the simulation training environment, as follows:

[0256]

[0257] Among them, f tra and g tra It can be obtained through simulation software.

[0258] S5. Reinforcement learning agent algorithm design, specifically:

[0259] S51. Initialize the reinforcement learning algorithm policy parameters θ, initialize the number of training iterations e = 0, and the total number of training iterations E = 500.

[0260] S52. Initialize the state space and reward function, and initialize the number of iterations k = 0, with a total number of iterations K = 200;

[0261] S53. The action space for the current time period is obtained based on the parameters of the traffic flow control strategy type, represented as:

[0262]

[0263] S54. Calculate the state space for the next time period based on the simulation training environment. and the reward function for the current time period

[0264] S55. Action space based on the current time period State space of the current time period The state space of the next time period Reward function for the current time period Update the algorithm policy. The reinforcement learning algorithm policy parameters are represented as follows:

[0265]

[0266] S56. Determine if the number of iterations is less than the total number of iterations K. If yes, proceed to step S53; otherwise, proceed to step S57.

[0267] S57. Determine if the number of training iterations is less than the total number of training iterations E. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy.

[0268] In step S6, the reinforcement learning agent is trained, specifically as follows:

[0269] S61. Deploy the simulation training environment built in step S4.

[0270] S62. Select different types of reinforcement learning agent algorithm policy parameter update functions Q for training. In this embodiment, Q-learning, DDPG, and A3C algorithms are used for training respectively.

[0271] S63. After training, select the reinforcement learning agent policy π* that has the best performance.

[0272] In step S7, the reinforcement learning agent is deployed.

[0273] S71. Deploy the best-performing reinforcement learning agent policy π* into the local information system.

[0274] S72. Based on the collected actual environmental conditions, calculate the optimal traffic flow control strategy for the actual environmental conditions, expressed as:

[0275] a * =π * (s real )

[0276] Where a* contains parameters for the dynamic lane management strategy; s real State variables containing two observation points.

[0277] Example 4

[0278] The scalable multi-strategy collaborative optimization method for highway traffic flow management provided in this embodiment differs from Embodiment 3 in that:

[0279] like Figure 3 As shown, the control strategy types in this embodiment include dynamic lane management and the newly added dynamic route guidance. Therefore, a corresponding dynamic lane management information board and a newly added dynamic route guidance information board are set up on the road segment.

[0280] S1. Define the reinforcement learning action space based on the control policy type, specifically as follows:

[0281] S11. Define the action space parameters involved in different types of control strategies, as shown in Table 4 below:

[0282] Table 4. Parameters of the action space involved in different types of control strategies

[0283]

[0284] S12. Identify the types of traffic flow control strategies that can be implemented on the road segment. In this embodiment, dynamic lane management and the newly added dynamic route guidance are included.

[0285] S13. Define the action space based on the types of traffic flow control strategies that can be implemented on the road segment and the parameters of different control strategies, as follows:

[0286]

[0287] S2. The reinforcement learning state space is defined based on the traffic flow observation state, specifically as follows:

[0288] S21. Define common state-space parameters, mainly including: cross-sectional average speed, cross-sectional average flow rate, road segment vehicle density, and road segment queue length;

[0289] S22. Organize the status observation points of the road segment and the status parameters that can be obtained from each observation point. This embodiment includes a total of four observation points, and the average cross-sectional speed and average cross-sectional flow can be obtained from all four observation points.

[0290] S23. Define the state space based on the state observation points of the road segment and the state parameters obtainable at each observation point, and add two new observation points, represented as follows:

[0291]

[0292] S3. Define reinforcement learning rewards based on the optimization objective, specifically as follows:

[0293] S31. Define the calculation methods for different target coefficients;

[0294] a) Formula for calculating the total travel time of vehicles on a road segment:

[0295]

[0296] b) Formula for calculating ramp queue length penalty factor:

[0297]

[0298] S33. L is not defined in this embodiment. O The reinforcement learning reward is defined based on the coefficients of various objectives, and is expressed as follows:

[0299]

[0300] This reward includes two parts: total vehicle travel time and ramp queue length penalty factor.

[0301] S4. Construct a simulation training environment, specifically as follows:

[0302] S41. Divide the simulated road segments according to the state space observation points and the scope of the control strategy implementation.

[0303] S42. Regarding the state space and action space, this embodiment uses a traffic flow mathematical model to construct the simulation training environment, as follows:

[0304]

[0305] Among them, f tra and g tra It can be obtained through simulation software.

[0306] S5. Reinforcement learning agent algorithm design, specifically:

[0307] S51. Initialize the reinforcement learning algorithm policy parameters θ, initialize the number of training iterations e = 0, and the total number of training iterations E = 500.

[0308] S52. Initialize the state space and reward function, and initialize the number of iterations k = 0, with a total number of iterations K = 200;

[0309] S53. The action space for the current time period is obtained based on the parameters of the traffic flow control strategy type, represented as:

[0310]

[0311] S54. Calculate the state space for the next time period based on the simulation training environment. and the reward function for the current time period

[0312] S55. Action space based on the current time period State space of the current time period The state space of the next time period Reward function for the current time period Update the algorithm policy. The reinforcement learning algorithm policy parameters are represented as follows:

[0313]

[0314] S56. Determine if the number of iterations is less than the total number of iterations K. If yes, proceed to step S53; otherwise, proceed to step S57.

[0315] S57. Determine if the number of training iterations is less than the total number of training iterations E. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy.

[0316] In step S6, the reinforcement learning agent is trained, specifically as follows:

[0317] S61. Deploy the simulation training environment built in step S4.

[0318] S62. Select different types of reinforcement learning agent algorithm policy parameter update functions Q for training. In this embodiment, Q-learning, DDPG, and A3C algorithms are used for training respectively.

[0319] S63. After training, select the reinforcement learning agent policy π* that has the best performance.

[0320] In step S7, the reinforcement learning agent is deployed.

[0321] S71. Deploy the best-performing reinforcement learning agent policy π* into the local information system.

[0322] S72. Based on the collected actual environmental conditions, calculate the optimal traffic flow control strategy for the actual environmental conditions, expressed as:

[0323] a * =π * (s real )

[0324] Where a* contains parameters for the dynamic lane management strategy; s real State variables containing two observation points.

[0325] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A scalable multi-strategy collaborative optimization method for highway traffic flow management, characterized in that, include: S1. Define the corresponding reinforcement learning action space based on different types of traffic flow control strategies; S2. Reinforcement learning state space based on traffic flow observation state definition; S3. Define the reward function for reinforcement learning based on different target coefficients; S4. Construct a simulation training environment; S5. Construct reinforcement learning agent strategies based on action space, state space, reward function, and simulation training environment; S6. Optimize the constructed reinforcement learning agent policy in a simulation training environment to obtain the optimal reinforcement learning agent policy; S7. Obtain the actual environmental state and calculate the optimal traffic flow control strategy based on the optimal reinforcement learning agent strategy; Step S5 includes: S51. Initialize the reinforcement learning algorithm policy parameters and initialize the number of training iterations; S52. Initialize the state space and reward function, and initialize the number of iterations; S53. Obtain the action space for the current time period based on parameters of traffic flow control strategy type; S54. Calculate the state space for the next time period and the reward function for the current time period based on the simulation training environment; S55. Update the algorithm policy parameters of the reinforcement learning algorithm based on the action space of the current time period, the state space of the current time period, the state space of the next time period, and the reward function of the current time period; S56. Determine if the number of iterations is less than the total number of iterations. If yes, proceed to step S53; otherwise, proceed to step S57. S57. Determine if the number of training iterations is less than the total number of training iterations. If yes, proceed to step S52; otherwise, end the process and obtain the reinforcement learning agent policy. In step S53, the action space for the current time period is obtained based on the parameters of the traffic flow control strategy type, as follows: in, Represents the action space of the current time period k; π θ This represents the reinforcement learning agent's strategy; Represents the state space of the current time period k; In step S55, the algorithm policy parameters for updating the reinforcement learning algorithm policy based on the action space of the current time period, the state space of the current time period, the state space of the next time period, and the reward function of the current time period are expressed as follows: Where θ represents the policy parameters of the reinforcement learning agent algorithm; Represents the state space for the next time interval k+1; Let Q represent the reward function for the current time period k; let Q represent the policy parameter update function of the reinforcement learning agent algorithm. Represents the state space of the current time period k; This represents the action space of the current time period k; G represents the reward function for the current time period k; tra This represents the function used to calculate the reward function.

2. The scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 1, characterized in that, Step S1 includes: S11. Obtain the parameters corresponding to different types of control strategies; S12. Obtain the types of traffic flow control strategies that can be implemented on the road segment; S13. Define the reinforcement learning action space based on the types of traffic flow control strategies that can be implemented on the road segment and the corresponding parameters.

3. The scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 2, characterized in that, The reinforcement learning action space in step S13 is represented as follows: in, Represents the action space; a 限速 Indicates parameters for dynamic speed limiting control strategy; a 硬路肩 Indicates the parameters of the hard shoulder opening control strategy; a 动态车道 Indicates parameters for dynamic lane management and control strategies; a 匝道 Indicates the parameters of the ramp flow regulation and control strategy; a 动态路径 Indicates the parameters of the dynamic path guidance and control strategy; a 其他 This indicates other control strategy parameters.

4. The scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 1, characterized in that, Step S2 includes: S21. Obtain the state parameters corresponding to the traffic flow observation state; S22. Obtain the status observation points of the road segment; S23. Define the reinforcement learning state space based on the state observation points and corresponding state parameters of the road segment.

5. A scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 4, characterized in that, The reinforcement learning state space in step S23 is represented as follows: in, Represents the state space; s1 represents the state parameters of the first observation point; s n s represents the state parameters of the nth observation point; N This represents the state parameters of the Nth observation point.

6. The scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 1, characterized in that, The different target coefficients in step S3 include the total travel time of vehicles on the road segment and the ramp queue length penalty factor. The total travel time for vehicles on this route is expressed as follows: Among them, L T T represents the total travel time of vehicles on a road segment; i represents a specific road segment; I represents the total travel time of vehicles on a road segment. all L represents the set of all road segments; i λ represents the length of road segment i; i ρ represents the number of lanes in road segment i; i (k) represents the traffic density of road segment i in time period k; I on Represents the set of road segments connecting to the entrance ramp; w i (k) represents the queue length of the entrance ramp of road segment i in time period k; The ramp queue length penalty factor is expressed as: Among them, L W σ represents the queue length penalty factor; i (k) represents the queue length penalty factor for segment i in time period k; The reward function for reinforcement learning in step S3 is expressed as: in, L represents the reward function; O These represent the coefficients of other objective functions.

7. The scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 1, characterized in that, The construction of the simulation training environment in step S4 is represented as follows: in, Describes the state space for the next time interval k+1; f tra The computation function representing the state space.

8. A scalable multi-strategy collaborative optimization method for highway traffic flow management according to claim 1, characterized in that, The optimal traffic flow control strategy for calculating the actual environmental conditions in step S7 is expressed as follows: a * =π * (s real ) Where a* represents the optimal traffic flow control strategy; π* represents the optimal reinforcement learning agent strategy; s real This indicates the actual environmental conditions.

Images