Variable speed limit active control method and system for expressway

Abstract

The invention provides an expressway variable speed limit active control method and system, and the method comprises the steps: building a mixed traffic flow scene considering the individuation of a driver, and setting scene parameters; acquiring traffic state data of the confluence road scene in real time; training an intelligent agent by using the obtained traffic state data as a state quantity to obtain a speed limit control optimization model; different training scenes are generated by Poisson distribution in the training process; an optimal speed limit value and a proper speed limit control position are selected in each control period T, a mobile bottleneck is generated by using a mobile bottleneck variable speed limit MVSL constructed based on an intelligent network connection vehicle, and the traffic condition is optimized from the two aspects of safety and traffic efficiency by actively constructing the mobile bottleneck.

Description

Technical Field

[0001] The invention belongs to the field of intelligent transportation technology, and in particular relates to a method and system for actively controlling variable speed limits on expressways. Background Art

[0002] Variable Speed ​​Limits is a traffic management control method commonly used in highway systems. The main principle is that the control center calculates different speed limit values ​​for different time periods based on traffic detection data, road conditions and other relevant influencing factors, and transmits the calculated speed limit values ​​to the variable speed limit signs on the highway. The speed limit values ​​of the main line of the highway are dynamically adjusted to increase the traffic flow rate, shorten vehicle travel time and reduce the accident rate.

[0003] However, the traditional control method also has many disadvantages. It can only monitor traffic flow data through fixed detectors, and its analysis of historical data cannot reflect the timeliness of traffic flow status. The driver's personalized reaction state to the driving strategy also has different degrees of influence on the effect of the implementation of his control strategy. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method and system for actively controlling a variable speed limit on a highway. The variable speed limit active control method of the present invention overcomes the problem that the speed limit value cannot be 100% obeyed, and is found to be more outstanding in terms of safety and traffic efficiency performance through testing.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for actively controlling a variable speed limit on a highway comprises the following steps: S1, build a mixed traffic flow scenario that takes into account the driver's personalization and set the scenario parameters; S2, obtaining traffic status data of the merging road scene according to the mixed traffic flow scene; S3, using the acquired traffic status data as the state quantity to train the intelligent agent to obtain the speed limit control optimization model; different training scenarios generated by Poisson distribution during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck is generated by using the mobile bottleneck variable speed limit MVSL built based on intelligent connected vehicles.

[0006] Preferably, according to the mixed traffic flow scenario, the density and flow rate of the merging area are obtained to draw a flow density map, and the flow density map is used as a basis for judging whether congestion occurs on the roads in the merging area.

[0007] Preferably, a road network perception module is used to obtain traffic status data of the entire road. The road network perception module includes an area detector, a point detector and a cloud control center. The detector is used to receive vehicle information and road status information, and upload it to the cloud control center; the cloud control center converts the list data into an array form and transmits it to the control model.

[0008] Preferably, the traffic status data includes vehicle travel paths, the number of lanes and traffic flow, and the vehicle travel paths include two paths: from the main road of the expressway to the main road and from the entrance ramp of the expressway to the main road.

[0009] Preferably, in each control cycle T Select the optimal speed limit value and appropriate speed limit control position: Find the leading intelligent connected vehicle in different lanes that meets the conditions at the starting point of the control area, and require the two intelligent connected vehicles to follow the fixed speed limit value Driving.

[0010] Preferably, the control model includes a hybrid reward function of traffic efficiency and safety; the expression is reward=w×V ̅+(1-w)×(TTC) ̅, where w represents the weight of traffic efficiency, V ̅ represents the average standardized speed in the bottleneck area, and (TTC)̅ represents the average standardized collision time of vehicles in the bottleneck area.

[0011] Preferably, two types of neural networks are constructed when training the control model, one of which is the critic Critic The Internet, the second is the actor's home Actors network, Critic The network is one; Actors The network is set to three, and each Actors The network activation function is changed to Tanh Function so that its action value output range is [-1, 1].

[0012] The road status information is used as Critic The input of the network, in each control cycle T Each independent agent relies on observing the local state to implement decision control; centralized critic The network uses the normalized advantage function estimation method to estimate a common action based on the reward value obtained by the action trajectory of each agent during centralized training; each agent i according to critic Network estimated state value input Actors The network selects the next control cycle T+1 Best Action

[0013] A variable speed limit active control system for a highway, comprising a scene building module, a traffic state acquisition module and a control module; The scenario building module is used to build mixed traffic flow scenarios that take into account the driver's personalization and set scenario parameters; A traffic status acquisition module, which acquires traffic status data of a merging road scene according to a mixed traffic flow scene; The control module uses the acquired traffic status data as the state quantity to train the control model, and trains the intelligent agent according to the real-time road traffic status to obtain the optimization model; different training scenarios are generated by Poisson during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck is generated by using the mobile bottleneck variable speed limit MVSL built based on intelligent connected vehicles.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for actively controlling the variable speed limit of a highway are implemented.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for actively controlling a variable speed limit on a highway.

[0016] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention provides a method for active control of variable speed limits on highways, which builds a mixed traffic flow scenario that takes into account the driver's personalization and sets scenario parameters; then obtains traffic state data of the merging road scenario according to the mixed traffic flow scenario; uses the acquired traffic state data as a state quantity to train a control model, and trains an intelligent agent according to the real-time road traffic state to obtain an optimized model; different training scenarios are generated by Poisson during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck MVSL built based on the intelligent connected vehicle is used to generate the mobile bottleneck. By actively constructing the mobile bottleneck, the traffic conditions are optimized from the aspects of safety and traffic efficiency.

[0017] The present invention considers only a few intelligent networked vehicles as the main control objects, and actively controls CAVs to form a mobile bottleneck as a means of speed control. The present invention adopts the proximal optimization strategy as the algorithm basis, and extends it to the direction of multi-agent reinforcement learning, and trains the speed control strategy according to the fully cooperative relationship model between multiple agents. When improving traffic performance, the present invention considers designing a hybrid reward function with the goal of optimizing traffic efficiency and ensuring road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1Proactively build speed limit control diagrams for mobility bottlenecks on smart highways; Figure 2 This is a schematic diagram of the composition information of the variable speed limit traffic flow in an example of the present invention; Figure 3 A schematic diagram of a basic diagram of traffic flow is constructed for the scenario of the embodiment of the present invention; Figure 4 This is a schematic diagram of the interaction between multi-agent reinforcement learning and the environment of the present invention; Figure 5 A schematic diagram of the change of reward value before and after variable speed limit control according to an embodiment of the present invention; Figure 6 Schematic diagram of average speed change in the bottleneck area before and after variable speed limit control in an embodiment of the present invention; Figure 7 Schematic diagram of average speed change distribution every 500m in the bottleneck area before and after the variable speed limit control in an embodiment of the present invention; Figure 8 Schematic diagram of average collision time of vehicles in bottleneck areas before and after variable speed limit control in an embodiment of the present invention.

[0019] Fig. 9 Schematic diagram of the process flow of the variable speed limit active control method for highways in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0021] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0022] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0023] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or a communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0029] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0030] Example 1 The present invention provides a method for actively controlling a variable speed limit on a highway, which is a method for actively controlling an intelligent networked vehicle to form a mobile bottleneck in a mixed traffic flow environment as a variable speed limit control method. The invention is implemented based on a multi-agent proximal optimization strategy algorithm, and uses multi-agent learning of driver driving behavior characteristics to be applied to intelligent networked vehicles to alleviate congestion in merging areas; specifically, the following steps are included: S1, build a mixed traffic flow scenario that takes into account the driver's personalization, and divide the vehicles into three categories according to the type of motor vehicle, named [HDV1, HDV2, HDV3], where HDV3 is a truck, HDV1 and HDV2 are cars. According to the driver's behavior characteristics, they are divided into conservative and aggressive types, where HDV1 is aggressive, HDV2 is conservative, and HDV3 is conservative; Figure 1 As shown, in the embodiment of the present application, the main lane of the traffic scene is set as a double lane, including a single entrance ramp, and the road is divided into a mobile speed limit control area, a merging area, and an area downstream of the merging area. Traffic flow formation part: In order to ensure the rationality of the scene, the traffic flow is divided into three stages, namely the warm-up stage, the increase stage, and the recovery stage. The traffic flow is divided into ramp entrance traffic flow and main lane traffic flow, and the traffic flow is sent according to the set time according to the Poisson distribution; the time set in this application is 600 time steps.

[0031] During training, 300 sets of random scenes are generated according to Poisson distribution, of which 250 sets are used for training.

[0032] The density and flow of the merging area are obtained to draw a flow density map, which is used as the basis for judging whether the roads in the merging area are congested. The following model of vehicles in the mixed traffic flow scenario is IDM, assuming that 80% of the vehicles are cars and 20% are trucks.

[0033] S2, acquiring the traffic status data of the entire road according to the mixed traffic flow scenario, wherein the traffic status data includes vehicle travel paths, number of lanes, and traffic flow.

[0034] The real-time occupancy rate of each lane in the bottleneck area and the average speed in the area. The real-time occupancy rate of each lane downstream of the bottleneck area and the real-time occupancy rate of the entrance ramp.

[0035] S3, constructs a mobile bottleneck variable speed limit MVSL, which is different from the traditional variable speed limit control that relies on traffic information signs. It forms a control area with intelligent connected vehicles as the control object and generates a mobile bottleneck.

[0036] S4, build a multi-agent control model, set hyperparameters, establish input state space, action space and reward function. The multi-agent control model adopts a fully cooperative control relationship, which is specifically divided into three agents, two of which are responsible for controlling the speed of the intelligent connected vehicle, and one agent is responsible for controlling the start and end positions of the speed limit control area, and using the mobile bottleneck variable speed limit MVSL to generate the mobile bottleneck. The algorithm training adopts a centralized training and decentralized decision-making framework. Specifically, the state of each agent's interactive shared perception with the environment constitutes a state set, which is sent to the critic Critic network for training. The Actor network corresponding to each agent makes different decision actions.

[0037] S5, training the multi-agent control model to obtain the optimal speed limit and appropriate control area: the perception data is input into the multi-agent training framework, and then the constructed control model is used to train the three agents to obtain the optimal speed limit and appropriate control area for each control cycle according to the real-time status of the mixed traffic flow. T The actions of different agents in the system are processed and converted into physical variables - speed and position, which are then published to the controlled CAVs to regulate the mixed traffic flow.

[0038] S6, setting up loading applications in the control model includes loading the trained parameter model and testing the effect of generating multiple mixed traffic flow scenarios. The test content includes two aspects: traffic efficiency and safety. The specific indicators are the average speed in the bottleneck area and the average collision time in the bottleneck area.

[0039] In S2, the vehicle travel paths include two routes: from the main road of the expressway to the main road, and from the entrance ramp of the expressway to the main road.

[0040] In S2, the bottleneck area section is the area where vehicles queue up in the upstream area of ​​the merging entrance. A roadside unit is arranged in each lane within 1 km of the merging area, at the ramp entrance, and in the downstream section of the merging area to obtain real-time occupancy information of each lane of the corresponding section.

[0041] In S2, the traffic status data of the entire road is obtained by using a road network perception module, which includes an area detector, a point detector and a cloud control center. The detector is used to receive vehicle information and road status information, and upload it to the cloud control center; the cloud control center converts the list data into an array form and transmits it to the control model.

[0042] In S3, the control area location is provided by the MAPPO-MVSL algorithm, including the starting point and the end point. The speed limit values ​​of the two intelligent network leading vehicles are also provided by the speed limit algorithm. As mentioned in S1, the main route consists of two lanes. Specifically, in each control cycle T The selection of the optimal speed limit value and the appropriate speed limit control position can be described as: finding the intelligent connected vehicle head vehicle in different lanes that meets the conditions at the starting point of the control area, and requiring the two intelligent connected vehicles to follow the fixed speed limit value. Driving.

[0043] In S3, the two intelligent connected vehicles maintain their original lanes and do not change lanes. When the controlled CAVs reach the end of the control area, the control of this group of CAVs can be released. If the road congestion is still not completely eliminated, continue to control the cycle. T The algorithm searches for suitable CAVs within the starting position of the control area provided by the algorithm until the congestion in the bottleneck area dissipates.

[0044] The state space includes the bottleneck area, the downstream of the bottleneck area, the real-time occupancy rate of each lane of the single entrance ramp, the average speed of the bottleneck area, and a discrete variable to determine whether the road is congested. The action space consists of the length of the speed control area and the speed limit value of the CAVs. ,in Indicates the action value regarding the speed limit, For the previous control cycle T-1 The speed limit value. It is a fixed value, which is the speed difference that can be increased or decreased each time.

[0045] The control model design includes a hybrid reward function for traffic efficiency and safety. The expression is , where represents the weight of traffic efficiency, represents the average normalized velocity in the bottleneck area, Represents the average standardized collision time of vehicles in the bottleneck area.

[0046] The S4 comprises the following steps: Step 4-1: Construct two types of neural networks when training the control model, one of which is the critic Critic The Internet, the second is the actor's home Actors network, Critic There is only one network, and Actors The network is determined by the number of multi-agents, here there are three. And each Actors The network activation function is changed to Tanh Function so that its action value output range is [-1, 1].

[0047] Step 4-2: Road status information as Critic The input of the network, in each control cycle T Each independent agent relies on observing the local state to implement decision control. critic The network uses the normalized advantage function estimation method to estimate a common action based on the reward values ​​obtained by the action trajectory of each agent during centralized training. i according to critic Network estimated state value input Actors The network selects the next control cycle T+1 Best Action .

[0048] In S4, the parameters of the control model include: number of iterations T =250; the discount factor is 0.99, the number of iterations performed at each policy update is 15, and the learning rate is 0.0005. Generalized Advantage Estimation of λ The parameter is 0.95, the number of agents is 3, the truncation parameter is 0.2, and the round step size for each update is 15.

[0049] In S4, the state space includes the real-time occupancy of each lane in the bottleneck area, the downstream of the bottleneck area, and the single entrance ramp, the average speed in the bottleneck area, and a discrete variable to determine whether the road is congested. This area describes the state of the MVSL, including the merging area, the downstream of the merging area, and the single entrance ramp area. The lane occupancy set of the three areas in each time step is defined as , and the average velocity in the confluence region is given by express, D It is used to determine whether the current environment is congested. It can be defined as:

[0050] In S4, It can be expressed as

[0051] in, , Is Actors Action values ​​generated by the network. ,in Indicates the action value regarding the speed limit, For the previous control cycle T-1 The speed limit value. is a fixed value, which is the speed difference that can be increased or decreased each time. is a two-dimensional vector that represents the start and end position of the mobile bottleneck area. The expression for the start or end position of the mobile bottleneck control can be written as:

[0052] in, is the maximum distance from the starting or ending control position, is the shortest distance to the starting or ending control position.

[0053] In S4, considering safety and traffic flow characteristics, the following two constraints are proposed for the application of the variable speed limit (MVSL) strategy: 1. To ensure safety and traffic efficiency, the optimal speed limit of the MVSL must be lower than the maximum value (80 km / h) and higher than the minimum value (30 km / h).

[0054] 2. For safe operation of the MVSL strategy, the speed difference between two consecutive time steps should not exceed 10 km / h.

[0055] In S4, the control model includes a hybrid reward function of traffic efficiency and safety. The expression is , where represents the weight of traffic efficiency, represents the average normalized velocity in the bottleneck area, Represents the average standardized collision time of vehicles in the bottleneck area.

[0056] Intelligent connected vehicles in mixed traffic flow have the characteristics of low latency and high reliability, are easy to control and respond quickly, and can improve traffic flow conditions in a timely manner, such as dissipating congestion waves in a short time. The present invention makes full use of the vehicle-road-cloud intelligent highway system, and the multi-agent reinforcement learning speed limit algorithm can significantly improve the control effect of highway vehicles, providing better theoretical and technical support for alleviating traffic congestion, improving traffic efficiency, and saving energy consumption.

[0057] The present invention can apply multi-agent deep reinforcement learning methods to variable speed limit traffic flow control, alleviate highway congestion in a more efficient manner, and significantly improve road traffic efficiency while ensuring driving safety.

[0058] The original traffic flow speed limit control method mainly relies on traffic information signs or fixed area speed limits, and lacks research on the personalized compliance of human drivers. The present invention proposes a new variable speed limit active control method to overcome the situation that human drivers cannot 100% comply with the speed limit value regulations, and after testing, it is found that it performs better in terms of safety and traffic efficiency performance.

[0059] Most technologies consider medium to high penetration rates when studying mixed traffic flow speed limit control, but it still takes time and is difficult to implement for intelligent connected vehicles in real road scenarios to reach this level. Therefore, this invention considers only controlling a small number of intelligent connected vehicles as the main object, and actively controlling the speed limit of vehicles to form a mobile bottleneck as a control method.

[0060] Most of the existing reinforcement learning control methods applied to the field of variable speed limit control are still single-agent, ignoring the advantages of cooperation between agents to jointly optimize the target. And most of them choose to use multi-agent reinforcement learning algorithms, which is difficult to guarantee training time and convergence. Therefore, this paper adopts the proximal optimization strategy as the algorithm basis, and extends it to the direction of multi-agent reinforcement learning, and trains the speed limit control strategy according to the fully cooperative relationship model between multi-agents.

[0061] Many invention studies only consider a single goal when considering improving road traffic conditions, such as only considering how to improve traffic efficiency or reduce environmental pollution, ensure safety, etc. Therefore, when improving traffic performance, the present invention considers designing a hybrid reward function with the goal of optimizing traffic efficiency and ensuring road safety.

[0062] Example 2 The embodiment of the present invention is a mobile bottleneck variable speed limit control method based on multi-agent reinforcement learning considering mixed traffic flow, such as Figure 1 As shown, the following steps are included.

[0063] Step 1: Build a mixed traffic flow network that takes into account the driver's personalization. The vehicle types are HDV and CAV. The detailed parameter information of different vehicles is shown in Table 1. The upstream area of ​​the main lane of the highway is 7.5 km long, the bottleneck area is up to 1 km long, the entrance ramp is 500 m long, and the downstream length of the bottleneck area is 2 km.

[0064] The traffic flow chooses to start according to two different traffic routes: the main road and the ramp. The specific start is every 600 seconds according to the road preheating stage, the increase stage, and the recovery stage. For detailed information on the number of departures and the proportion of departure types, see Figure 2 , whether to build congestion scenes and basic maps of congested areas can be found in Figure 3 .

[0065] Table 1 Definition of personalized vehicle parameters

[0066] Step 2: The information perception module consists of a point detector, an area detector, and a traffic control unit, which is used to obtain the number of lanes, speed limit value, traffic path, traffic flow, and real-time occupancy rate of each lane in the bottleneck area and downstream sections and single entrance ramps, and the average speed in the bottleneck area.

[0067] Step 2-1: Divide the highway section with the entrance ramp into several areas. The main lane at the entrance of the ramp is the bottleneck area. The detector receives all vehicle information and traffic status information passing through the area described in step 1 of the current main road, and sends it to the cloud control center. The collected data includes real-time occupancy rate and average vehicle speed.

[0068] Step 2-2: Obtain traffic status data. Roadside units are specifically set up in three locations, namely the bottleneck area, the ramp entrance, and the downstream of the bottleneck area. By deploying detectors in each lane in the bottleneck area, the downstream of the bottleneck area, and the ramp entrance, real-time traffic flow data can be obtained.

[0069] Step 2-3: Convert the data format of traffic status. The control unit is a computer with independent traffic control algorithm. The traffic control unit stores the data in the form of an array and transmits it to the multi-agent training module.

[0070] Step 3: Build a multi-agent training module. Figure 4 , each agent can be controlled according to the current control cycle T Lower Observation Value and the next control cycle T+1 Internal adoption basis Select Action , the observed value of the next control cycle Each independent agent obtains a trajectory to calculate the state value , calculate the advantage function . And the trajectory data { }Stored in the experience replay pool D. Update the neural network as many times as needed K , select small batches from the experience replay pool for each agent i Calculate the Actor network gradient and the Critic network gradient, and apply gradient ascent and gradient descent to update the network parameters respectively.

[0071] Step 4: Construct the MAPPO-MVSL algorithm Step 4-1: MAPPO-MVSL traffic flow control constraint: The two CAVs do not change lanes in the control area. If one of the CAVs reaches the merging area, the control of the group of CAVs will stop. If the road congestion persists, the CAVs will be re-controlled to form a mobile bottleneck at the control starting point.

[0072] Step 4-2: Set MAPPO parameters. See Table 2 for the specific design. Figure 1As shown in Figure 1, the longitudinal distance and constant speed of two CAVs (TV and OV) are controlled. A moving bottleneck is constructed by selecting CAVs in each lane, and these CAVs travel at different fixed speeds. The longitudinal distance between the two CAVs provides space for the HDVs in the original lane to change lanes, allowing them to cross the moving bottleneck and travel at a free flow speed in the target lane.

[0073] Step 4-3: State space: In order to fully describe the entire environment affected by MVSL, the traffic scene is divided into several areas to describe the state of MVSL, including the merging area, the downstream area of ​​the merging area, and the single entrance ramp area. t The real-time lane occupancy of the three inner zones is defined as , and the average velocity in the confluence region is given by express, D It is used to determine whether the current environment is congested. It can be defined as:

[0074] Step 4-4: Action Space: Considering that the MVSL controller consists of multiple agents, the action It can be expressed as

[0075] in, . is the action value generated by the Actor network. The speed of the two controlled CAVs and the control length of the mobile bottleneck are three key factors affecting the traffic flow. Therefore, the set It is designed to indicate the speed limit of the leading CAV in one lane and the following CAV in another lane. The action determines the speed limit according to the formula: ,gather is a two-dimensional vector that represents the start and end position of the control area. The expression for the start or end position of the mobile bottleneck control can be written as: in, is the maximum distance from the starting or ending control position, is the shortest distance to the starting or ending control position.

[0076] Considering safety and traffic flow characteristics, the following two constraints are proposed for the application of the variable speed limit (VSL) strategy: 1. To ensure safety and traffic efficiency, the optimal speed limit for VSL must be lower than the maximum value (80 km / h) and higher than the minimum value (30 km / h).

[0077] 2. For safe operation of the VSL strategy, the speed difference between two consecutive time steps should not exceed 10 km / h.

[0078] Step 4-5: The reward function ensures that the agent is encouraged or inhibited to perform specific actions according to the environment requirements, thereby guiding them to effectively solve the traffic problem. In order to improve traffic efficiency, the MVSL controller uses the average normalized speed in the bottleneck area as part of improving traffic efficiency, considering the current time step t The average speed of all vehicles in the bottleneck area. Considering safety, a method for calculating the time to collision (TTC) is proposed. i At time step t The time to collision (TTC) can be expressed as: (5) Each time step t If each vehicle i The TTC is less than the set threshold , then the vehicle will be included in the calculation of the safety reward function i of TTC.

[0079]

[0080] Indicates that TTC is less than the threshold in the bottleneck area The number of vehicles. Control cycle T Average standardization within The calculation is as follows:

[0081] The goal of the MVSL controller is to maximize traffic efficiency while ensuring road safety. Therefore, the hybrid reward function can be expressed as:

[0082] Traffic efficiency weight Set to 0.7.

[0083] S5, input the perception information into the multi-agent training module, and train the multi-agent by constructing the MAPPO-MVSL algorithm; train the agent according to the real-time traffic status, and train the MAPPO-MVSL method according to different training scenarios to obtain the control cycle of each control cycle. T The optimal speed limit value and appropriate speed limit control location are determined. At the same time, the cloud control center sends the information to the controlled CAVs to form a mobile bottleneck. The training process includes the following steps: Step 5-1: For all agents u , initialize the Critic network and Actors Parameters .

[0084] Step 5-2: Use the current strategy Parameters ← , and initialize the target Critic Parameters ← .

[0085] Step 5-3: Initialize the experience buffer D; Step 5-4: For iteration i = 1, 2, ..., L; this step is the main loop of the outer layer, which usually represents the main iteration number of the entire training process. Each iteration i usually contains a complete policy evaluation and update process. This is the highest level loop of the algorithm.

[0086] Step 5-41: Initialize state .

[0087] Step 5-42: For an episode t = 1, 2, ..., T; in each episode, the agent selects actions based on the current strategy, observes environmental feedback, and collects experience data (state, action, reward, next state, etc.). T is the number of time steps in a single episode, representing the number of times the agent acts in the environment.

[0088] Step 5-421: Each agent u According to the current strategy Perform an action.

[0089] Step 5-422: Get Rewards and the next state .

[0090] Step 5-5: For each agent u Get track ; Step 5-6: Calculation .

[0091] Step 5-7: Calculate the advantage function ; Step 5-8: Transform the data into } is stored in the experience buffer D.

[0092] Step 5-9: For k≡ 1, 2, ..., K; K is the number of batches of parameter updates, indicating that multiple parameter updates are performed with the same batch of data in each major iteration to improve the stability and efficiency of the algorithm. This is a further refinement loop in each major iteration, usually used to optimize model parameters. After collecting enough empirical data, this loop is used to update the parameters of the policy network and the value network multiple times.

[0093] Step 5-91: Shuffle and renumber the data in order.

[0094] Step 5-92: For j = 0, 1, ..., -1; Step 5-921: Select data group .

[0095] Step 5-922: for u=1, 2, ..., N; Step 5-9221: ; Step 5-9222: ; Step 5-9223: Using the Adam optimizer, apply gradient ascent updates ; Step 5-9224: Using the Adam optimizer, apply gradient descent updates .

[0096] Test example The experiment was conducted in SUMO, and the scenario was the M25 motorway in the UK. The road includes a control area, a bottleneck area, a single entrance ramp area, and a downstream area. The upstream control area is 7.5 km long, the bottleneck area is 1 km long, the entrance ramp area is 500 m long, and the downstream area is 2 km long.

[0097] Considering the diversity of vehicles, the vehicle types are defined individually. Human-driven vehicles (HDVs) are divided into three different types, while autonomous vehicles (CAVs) are divided into one type. The lane-changing model uses the default LC2013 model in SUMO. The longitudinal following behavior of CAVs follows the IDM. In the simulation, 80% of the vehicles are cars with a length of 3.5 meters, while 20% of the vehicles are trucks with a length of 7 meters.

[0098] Based on the 24-hour traffic flow data from February 25 to March 25, 2024, the test will focus on the one hour between 17:00 and 18:00 after get off work, that is, 3600 seconds. In order to simulate real traffic conditions, the traffic flow input will be updated every 600 seconds. The specific values ​​of the vehicles introduced are shown in Figure 2As shown in Figure 1, it includes mainline traffic and ramp traffic. The mainline traffic is composed of CAVs and HDVs. The on-ramp traffic is mainly composed of HDVs.

[0099] The proposed MAPPO-MVSL strategy adjusts the speed limit in increments of 10 km / h. The speed limit values ​​include: [30, 40, 50, 60, 70, 80] km / h. In order to adjust the appropriate training process, the hyperparameters must be adjusted accordingly. The hyperparameter configuration for the MAPPO-MVSL strategy is shown in Table 2. MAPPO-MVSL and DDPG-MVSL were trained for 250 rounds respectively, and the training reward curves are shown in Figure 5 As shown in the figure, the reward value fluctuates around 42.5 during the training process.

[0100] Table 2 MAPPO hyperparameter settings

[0101] exist Figure 6 In the example, the traffic scenario is severely congested in the merging area. MAPPO-MVSL shows more efficient optimization in the bottleneck area, with less fluctuation in speed change during the control cycle. The histogram on the right shows that MAPPO-MVSL improves traffic efficiency by 40% compared to DDPG-MVSL. For further analysis, Figure 7 The heat map of the speed within every 100 meters in the bottleneck area is shown as it changes over time. It can be seen that MAPPO-MVSL significantly improves the average speed of different sections in the bottleneck area.

[0102] exist Figure 8 In the traffic situation, a large number of vehicles merging from the entrance lane into the main road affected the normal traffic in the inner lane of the main line, and frequent stop-and-go situations and high collision risks occurred. MAPPO-MVSL showed better safety performance, ensuring that the average collision time of each vehicle increased by about 16%.

[0103] The embodiment of the present invention describes a method for actively constructing speed limit control based on multi-agent deep reinforcement learning of mobile bottlenecks considering mixed traffic flow. If implemented in the form of a software function module and sold or used as an independent product, the method can be stored in a computer-readable storage medium.

[0104] In yet another embodiment of the present invention, there is provided a highway variable speed limit active control system, comprising a scene building module, a traffic status acquisition module and a control module; The scenario building module is used to build mixed traffic flow scenarios that take into account the driver's personalization and set scenario parameters; A traffic status acquisition module, which acquires traffic status data of a merging road scene according to a mixed traffic flow scene; The control module uses the acquired traffic status data as the state quantity to train the control model, and trains the intelligent agent according to the real-time road traffic status to obtain the optimization model; different training scenarios are generated by Poisson during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck is generated by using the mobile bottleneck variable speed limit MVSL built based on intelligent connected vehicles.

[0105] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the variable speed limit active control method of the highway.

[0106] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the variable speed limit active control method for highways in the above embodiment.

[0107] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0108] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for actively controlling a variable speed limit on a highway, characterized in that: The following steps are included: S1, build a mixed traffic flow scenario that takes into account the driver's personalization and set the scenario parameters; S2, obtaining the merging road scene according to the mixed traffic flow scene, including the traffic status data of the entrance ramp and the downstream of the main road; S3, using the acquired traffic status data as the state quantity to train the intelligent agent to obtain the speed limit control optimization model; different training scenarios generated by Poisson distribution during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck is generated by using the mobile bottleneck variable speed limit MVSL built based on intelligent connected vehicles.

2. The method for active control of variable speed limit on a highway according to claim 1, characterized in that: According to the mixed traffic flow scenario, the density and flow rate of the merging area are obtained to draw a flow density map, which is used as the basis for judging whether there is congestion on the roads in the merging area.

3. The method for active control of variable speed limit on a highway according to claim 1, characterized in that: The road network perception module is used to obtain the traffic status data of the entire road. The road network perception module includes an area detector, a point detector and a cloud control center. The detector is used to receive vehicle information and road status information, and upload it to the cloud control center; the cloud control center converts the list data into an array form and transmits it to the control model.

4. The method for active control of variable speed limit on a highway according to claim 1, characterized in that: The traffic status data includes vehicle travel paths, lane numbers and traffic flow, and the vehicle travel paths include two routes: from the main road of the expressway to the main road and from the entrance ramp of the expressway to the main road.

5. The method for active control of variable speed limit on a highway according to claim 1, characterized in that: In each control cycle T Select the optimal speed limit value and appropriate speed limit control position: Find the leading intelligent connected vehicle in different lanes that meets the conditions at the starting point of the control area, and require the two intelligent connected vehicles to follow the fixed speed limit value Driving.

6. The method for actively controlling a variable speed limit on a highway according to claim 1, characterized in that: The control model includes a hybrid reward function that considers both traffic efficiency and safety; the expression is reward=w×V ̅+(1-w)×(TTC)̅, where w represents the weight of traffic efficiency, V ̅ represents the average normalized speed in the bottleneck area, and (TTC) ̅ represents the average normalized collision time of vehicles in the bottleneck area.

7. The method for actively controlling a variable speed limit on a highway according to claim 1, characterized in that: Two types of neural networks are constructed when controlling model training, one of which is the critic Critic The Internet, the second is the actor's home Actors network, Critic The network is one; Actors The network is set to three, and each Actors The network activation function is changed to Tanh Function, so that its action value output range is [-1, 1]; The road status information is used as Critic The input of the network, in each control cycle T Each independent agent relies on observing the local state to implement decision control; Centralization critic The network uses the normalized advantage function estimation method to estimate a common action based on the reward value obtained by the action trajectory of each agent during centralized training; each agent i according to critic Network estimated state value input Actors The network selects the next control cycle T+1 Best Action 8. A variable speed limit active control system for a highway, characterized in that: Including scene building module, traffic status acquisition module and control module; The scenario building module is used to build mixed traffic flow scenarios that take into account the driver's personalization and set scenario parameters; Traffic status acquisition module, which acquires traffic status data of merging roads (including entrance ramps and downstream of the main road) according to mixed traffic flow scenarios; The control module uses the acquired traffic status data as the state quantity to train the intelligent agent to obtain the speed limit control optimization model; different training scenarios are generated by Poisson distribution during the training process; in each control cycle T The optimal speed limit value and the appropriate speed limit control position are selected, and the mobile bottleneck is generated by using the mobile bottleneck variable speed limit MVSL built based on intelligent connected vehicles.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for active control of variable speed limit on a highway as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for active control of variable speed limit on a highway as claimed in any one of claims 1 to 7 are implemented.

Images