Variable speed-limiting control method for confluence area in highway road-alarm linkage mode

Abstract

The invention discloses a variable speed-limiting control method for a confluence area in a highway road-alarm linkage mode, and relates to the technical field of variable speed-limiting control for the confluence area, and the method specifically comprises the following steps: firstly, constructing a lane-level variable speed-limiting control model based on reinforcement learning; secondly, safety constraint and intelligent control are conducted, and it is guaranteed that the output end speed limit value meets the condition that the vehicle speed change of adjacent lanes in the same time period is not higher than the preset speed per hour and the vehicle speed change of the same lane in the adjacent time periods is not higher than the preset speed per hour; in the aspect of variable speed limit control of the confluence area, the traffic efficiency and safety of the highway confluence area are remarkably improved. Through a reinforcement learning-based DVSL control model, an optimized action space design and a perfect safety constraint and inter-lane safety detection mechanism, a new technical path and a solution are provided for the development of an intelligent traffic system.

Description

Technical Field

[0001] The present invention belongs to the technical field of variable speed limit control in a merging area, and in particular is a variable speed limit control method in a merging area in a highway road police linkage mode. Background Art

[0002] The merging area of ​​highways is one of the common areas of traffic bottlenecks. Especially when the traffic demand is large, the merging area often causes traffic conflicts, traffic shocks and driving speed reduction. Generally speaking, after entering the acceleration lane from the ramp, vehicles need to perform a series of complex operations, including acceleration, selecting an acceptable gap and changing lanes to merge with the main line traffic. These operations not only increase the operating burden of the driver, but also easily lead to traffic conflicts, thus affecting the normal operation of the main line traffic.

[0003] Traffic management in merge areas has always been a research focus at home and abroad. Foreign countries such as the United States, Germany, and Japan have made significant progress in intelligent transportation systems and vehicle-road cooperative technologies, and have verified the effectiveness of variable speed limits in alleviating traffic congestion and reducing traffic accidents through actual deployment and large-scale testing. For example, variable speed limit technology is widely used in the United States' Intelligent Transportation System (ITS), which dynamically adjusts speed limits based on real-time traffic flow data to improve road capacity and safety.

[0004] In China, with the development of intelligent transportation systems, more and more research institutions and universities have begun to pay attention to the issue of traffic management in merging areas. Institutions such as Tsinghua University and Tongji University have conducted a lot of research on vehicle-road collaboration and intelligent transportation systems, developed a variety of traffic management systems based on V2X communication technology, and tested and verified them on actual roads.

[0005] In order to ease the traffic pressure in the merging area, optimize the traffic flow, and improve the traffic efficiency of the highway, this application proposes a variable speed limit control method for the merging area under the highway road police linkage mode. Based on the differences in the traffic flow conditions of each lane on the main line of the merging area affected by merging vehicles, this method uses a reinforcement learning algorithm to propose a lane-level variable speed limit control model.

[0006] The variable speed limit control method for the merging area proposed in this application dynamically adjusts the speed limit of each lane according to real-time traffic data and lane-level traffic flow characteristics through a reinforcement learning algorithm. This differentiated speed limit strategy (DVSL) can better adapt to the complex traffic environment of the merging area, effectively reduce traffic conflicts, alleviate traffic shocks, and improve the operating efficiency of the main line traffic. In practical applications, this method combines the road-police linkage mode, and further improves the intelligence level and response speed of traffic management through real-time monitoring and dynamic adjustment. In summary, based on the research results at home and abroad, this application innovatively proposes a lane-level variable speed limit control model based on reinforcement learning, which provides new ideas and methods for solving the traffic management problems in the merging area of ​​highways, and has important theoretical significance and practical application value. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a variable speed limit control method for a merging area under a highway road police linkage mode.

[0008] A variable speed limit control method for a merging area under a highway road police linkage mode, the method specifically comprising:

[0009] First, a lane-level variable speed limit control model based on reinforcement learning is constructed as follows:

[0010] St1, set the corresponding parameters, including the maximum number of simulation rounds N, the simulation step size T per round, the soft update rate τ, the ε decay rate k0 per round, and the minimum ε value ε min ; Initialize the PER-DDQN agent local Q network parameters θ and target Q network parameters Let n = 0, t = 0;

[0011] St2, number of simulation rounds n = n + 1;

[0012] St3, simulation time step t=t+1;

[0013] St4, at the t time step, the SUMO simulation software converts the state information S collected by the sensors on the road into t Passed to the PER-DDQN agent;

[0014] St5, the agent uses the ε-greedy strategy to select the corresponding action, that is, randomly select an action from the action space with a probability of ε, or output an action according to the state with a probability of 1-ε

[0015] St6, use the safety detection mechanism to regulate, and convert the output action into a speed limit value and pass it to the speed limit area vehicle, then enter the next control cycle, and the agent obtains a new state S t+1And reward r t ;

[0016] St7, the experience track (S t , a t , S t+1 , r t ) is stored in the experience replay pool, and the samples extracted by priority experience replay are used to update the local Q network parameters θ using gradient descent according to formula (9), and soft update is used Update target network parameters

[0017] St8, if t<T, return to St3, otherwise go to St9;

[0018] St9,ε=min(ε*k0,ε min ), if n<N, return to St2, otherwise go to St10;

[0019] St10, the process ends;

[0020] Secondly, safety constraints and intelligent control are performed to ensure that the speed limit value at the output end is met, and the speed change of adjacent lanes in the same period does not exceed the preset speed 1; the speed change of adjacent lanes in the same period does not exceed the preset speed 2;

[0021] Finally, human assistance is introduced. Under the combination of human assistance with real-time monitoring and dynamic adjustment mechanism, it is ensured that the variable speed limit control method in the merging area of ​​the highway can quickly respond to changes in traffic conditions and provide immediate traffic guidance and management.

[0022] Furthermore, the preset speed 1 is 20 km / h, and the preset speed 2 is 10 km / h.

[0023] Furthermore, the state space needs to be defined before the lane-level variable speed limit control model is constructed. The specific method is as follows:

[0024] The state space is defined to contain 19 states, as follows:

[0025]

[0026] In the formula, S is the state set, R up is the occupancy rate of each upstream lane, R down is the occupancy rate of each downstream lane, R bottleneckup is the occupancy rate of each lane upstream of the bottleneck area, R ramp is the occupancy rate of the upstream lane of the ramp, is the average speed of each lane upstream of the bottleneck area, is the average speed of each lane downstream of the bottleneck area, V limitIt is the set of speed limit values ​​for each lane in the control area; the traffic flow information in the above-mentioned merging area is collected by detectors set at corresponding positions.

[0027] Furthermore, after defining the state space, the reward function needs to be constructed using the total travel time and the merging area travel time; specifically:

[0028] f r =α1f TTS-merge +α2f TTS ;

[0029] In the formula, f r is the comprehensive reward function; α1 and α2 are the preset weight coefficients, f TTS-merge Construct a reward function for the merge area travel time; f TTS is the reward function related to the total travel time.

[0030] Furthermore, f TTS-merge Specifically:

[0031]

[0032] In the formula, f TTS-merge is the reward function related to the travel time in the merging area, k is the vehicle number passing through the merging area, and K is the number of vehicles passing through the merging area in one time step.

[0033] Furthermore, f TTS Determined by:

[0034] First, the relationship between the inflow and outflow of the road network and the vehicle's full travel time is constructed as follows:

[0035]

[0036] Where Z is the total time compensation, T is the time interval, and F in F is all vehicles flowing into the road network. out All vehicles leaving the road network;

[0037] because and Therefore, the reward function for the total travel time is designed as follows:

[0038] f TTS =f out -f in ;

[0039] In the formula, f TTS is the reward function related to the total travel time, f in is the number of vehicles flowing into the road network within a time step; f out is the number of vehicles flowing out of the road network in one time step.

[0040] Furthermore, the specific methods of safety constraints and intelligent control are as follows:

[0041] First, the output speed limit value must meet the following safety constraints: the speed change of adjacent lanes in the same period is not higher than 20km / h; the speed change of adjacent lanes in the same period is not higher than 10km / h;

[0042] Introduce a lane-to-lane safety detection mechanism, specifically:

[0043] After obtaining the current speed limit value, the intelligent transportation system outputs the speed limit change value while meeting the safety constraints, automatically calculates the new round of speed limit values, and determines whether it is feasible. If not, repeat this step. If it is feasible, automatically calculate the next round of speed limit values.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This application significantly improves the traffic efficiency and safety of highway merging areas in terms of variable speed limit control in merging areas. Through the DVSL control model based on reinforcement learning, optimized action space design, and perfect safety constraints and lane-to-lane safety detection mechanism, it provides a new technical path and solution for the development of intelligent transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the speed limit control process of the present invention;

[0047] Figure 2 This is a schematic diagram of the speed limit change constraint in adjacent time periods of the same lane of the present invention;

[0048] Figure 3 This is a schematic diagram of the speed limit change constraints of adjacent lanes in the same period of time according to the present invention;

[0049] Figure 4 This is a schematic diagram of the detector position of the present invention;

[0050] Figure 5 This is a schematic diagram of the action space design of the present invention;

[0051] Figure 6 This is a control flow chart of the safety detection mechanism of the present invention;

[0052] Figure 7 It is a schematic diagram of the lane-level variable speed limit control model framework based on the improved DDQN of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The present application provides a method for controlling variable speed limits in a merging area in a highway police linkage mode, such as Figure 1 As shown in the figure, a variable speed limit control system is first set up. In this system, by setting detection points and placing detectors in the road network, traffic status information, including vehicle density, speed, flow and other data, can be collected in real time. These data are transmitted to the intelligent control system through sensors, so that it can effectively determine the speed limit strategy for each lane after calculation and transmit it to the road network. After the vehicle receives the speed limit control information, the driver will adjust the speed according to the speed limit information to ensure a safe distance and appropriate speed in the merging area. Through this intelligent speed limit control system, the appropriate speed limit strategy will be dynamically determined based on the real-time collected traffic status information and intelligent control algorithms, and the traffic flow will be optimized and regulated in the merging area, the stability and traffic efficiency of the traffic flow will be improved, and the probability of traffic accidents will be reduced, thereby achieving safer and more efficient highway operation management.

[0055] After the vehicle has the intention to change lanes, it needs to fully perceive and evaluate the surrounding environment. Only when safety is ensured can the lane change operation be performed. In order to ensure the safety of vehicle lane changes, maintain the stable operation of traffic flow and minimize the risk of accidents, this study further refines the speed limit rules in combination with relevant literature, requiring the intelligent control system to meet basic speed limit constraints when outputting lane-level speed limit actions.

[0056] 1) Basic safety constraints

[0057] In order to ensure the rationality of the implementation of the speed limit control strategy, the vehicle should travel within the range specified by the speed limit regulations. According to the speed limit regulations of my country's expressways, the maximum speed limit is set to 120km / h, and the speed limit shall not be lower than 60km / h to ensure the basic requirements of convenient and fast expressways. Therefore, it is stipulated that the speed limit output by the system must meet the basic safety constraints, as shown in the following formula.

[0058] 60km / h≤v i,t ≤120km / h

[0059] Where v i,t It is represented by the speed limit value output by the i-th lane at time step t, km / h. Lane 1 represents the outermost lane close to the ramp, lane 2 represents the middle lane, and lane 3 represents the innermost lane.

[0060] 2) Constraints on speed limit changes during adjacent time periods

[0061] like Figure 2 As shown in the figure, considering the adverse effects of vehicle speed changes between speed limit controls in adjacent periods of the same lane on driving safety, the design of the speed limit control system needs to take into account the stability of traffic flow; sudden speed changes may cause emergency braking or acceleration between vehicles, increasing the risk of traffic accidents; by adopting a smooth transition speed limit adjustment method, the system can reduce the speed difference between vehicles, allowing drivers to adapt to the new speed limit environment more easily, thereby reducing the possibility of traffic accidents; combined with previous speed limit value research, a safety constraint is defined that the speed change in adjacent periods of the same lane is not higher than 10km / h, as shown in the following formula.

[0062] |v i,t -v i,t-1 |≤10km / h

[0063] 3) Constraints on speed limit changes in adjacent lanes;

[0064] A large speed difference between adjacent lanes during the same period may also have an adverse effect on driving safety. If the speed limits of adjacent lanes differ too much during the same period, it will be difficult for the driver to change lanes during driving, increasing the instability of traffic flow and the risk of collision between vehicles. Therefore, when designing a speed limit control system, the coordination between adjacent lanes is taken into account. In order to maintain the stability of the overall traffic flow and the safety of the driver's lane change, the speed limit value combination output by the intelligent control system is defined to meet the requirement that the speed change of adjacent lanes during the same period is not higher than 20km / h, as shown in the following formula;

[0065] |v i,t -v i-1,t |≤20km / h(i=2,3)

[0066] The purpose of this study is to control the speed limit of each lane upstream of the merging area through reinforcement learning methods, thereby improving traffic flow stability. For the DVSL control problem, the state space contains information that can reflect the real-time traffic conditions, and the action space refers to the speed limit adjustment value output by the intelligent control system. During the learning process, the agent obtains the state S of itself and its environment at time t. t , and take action a according to strategy π t , the environment is in action a t The agent is fed back a numerical reward r t+1 and a new state s t+1 , and repeat this process until it terminates;

[0067] In reinforcement learning problems, the state space represents all possible environmental conditions. When solving the variable speed limit control problem with reinforcement learning, the state space is usually defined as the traffic flow conditions in the merging area. The parameters commonly used to measure traffic flow conditions are mainly vehicle density, speed, and lane occupancy. In order to accurately describe the environmental conditions of the merging area, the occupancy or average speed of each lane in the upstream, downstream, upstream of the ramp, and bottleneck area of ​​the merging area is selected to form the state space.

[0068] In addition, when designing the state space, this study also incorporates the current speed limit of each lane into the state space so that the agent can choose the corresponding action within the safety constraint according to the current speed limit. Figure 4 shown.

[0069] Combined with the detector position, the state space is defined to contain 19 states, as shown below.

[0070] In the formula, S is the state set, R up is the occupancy rate of each upstream lane, R down is the occupancy rate of each downstream lane, R bottleneckup is the occupancy rate of each lane upstream of the bottleneck area, R ramp is the occupancy rate of the upstream lane of the ramp, is the average speed of each lane upstream of the bottleneck area, is the average speed of each lane downstream of the bottleneck area, V limit It is the set of speed limit values ​​for each lane in the control area; the traffic flow information in the merging area collected by these position detectors can accurately reflect the traffic conditions in the merging area within the control period, which is conducive to the intelligent agent to make reasonable speed limit actions according to the current environmental status.

[0071] In the past, when designing the action space of the lane-level variable speed limit model based on reinforcement learning, the speed limit value of each lane in the control area was usually used. For highways, the speed limit range is generally 60-120km / h. According to the "Highway Speed ​​Limit Sign Design Specifications", the speed limit value should be an integer multiple of 10km / h. Therefore, the action space of each lane is {60, 70, 80, 90, 100, 110, 120}km / h. There are 343 possible speed limit value combinations for 3 lanes. The action space is too large, and the problem of large amount of calculation and difficulty in solving will be faced in the process of solving using reinforcement learning; in addition, the speed difference between adjacent lanes may exceed 20m / h in dangerous scenarios, increasing the risk of collision when the driver changes lanes.

[0072] In order to meet the safety constraints (Equations 3-14 and 3-15), reduce the dimension of the action space, and combine the safety constraint that the speed limit values ​​between adjacent lanes should not differ by more than 20 km / h, the possible speed limit combinations are reduced to 125. However, the action space after preliminary optimization is still very large, which makes training more difficult. In addition, it is difficult to ensure the safety constraint that the speed limit difference between two adjacent control lanes in the same lane does not exceed 10 km / h during the action selection process. Therefore, this study sets the action space of each lane to the relative speed change value, that is, the action output by the agent is {-10, 0, +10} km / h. The action space has only 27, such as Figure 5 The above method can effectively reduce the dimension of action space, greatly reduce learning cost and improve learning efficiency.

[0073] The purpose of setting the reward function is to concretize and quantify the task objectives, further determine the learning direction of the agent, and directly affect the convergence speed and final performance of the algorithm. In order to achieve the goal of using variable speed limit strategies to alleviate congestion, reduce travel time, and thus improve the efficiency of road network operation, in the process of designing the reward function, this study considered two indicators, the full travel time and the travel time in the merging area, on the basis of improving the efficiency and safety of the merging area. This is because if the reward function is constructed only by minimizing the travel time in the merging area, when the traffic flow is too large, it may restrict the inflow of upstream vehicles and cause queuing in the upstream of the merging area; on the contrary, if the minimization of the full travel time is used as the optimization control goal, it may cause the merging area to change to a low-speed and high-density state, thereby increasing the risk of accidents. Therefore, this study comprehensively considers these two situations and uses the full travel time and the travel time in the merging area to jointly construct the reward function.

[0074] 1) Travel time in the merging area

[0075] In order to encourage the agent to develop a strategy that enables the vehicle to pass through the merging area in the shortest time, the reward function related terms are constructed according to the travel time in the merging area (total time spent, TTS) as shown in the following formula.

[0076]

[0077] f TTS-merge is the reward function related to the travel time in the merging area, k is the vehicle number passing through the merging area; K is the number of vehicles passing through the merging area in one time step;

[0078] 2) Total journey time

[0079] In view of the fact that it is difficult to detect the vehicle's entire travel time within one control cycle, according to the prior art, the inflow and outflow of the road network are directly related to the vehicle's entire travel time. The relationship between the inflow and outflow of the road network and the vehicle's entire travel time is as follows.

[0080]

[0081] Where Z is the total time compensation, T is the time interval, and F in F is all vehicles flowing into the road network. out All vehicles leaving the road network;

[0082] Apparently and Therefore, the reward function for the total travel time is designed as follows:

[0083] f TTS =f out -f in ;

[0084] In the formula, f TTS is the reward function related to the total travel time, f in is the number of vehicles flowing into the road network within a time step; f out is the number of vehicles flowing out of the road network in one time step;

[0085] 3) Comprehensive reward function

[0086] The comprehensive reward function f of the model is designed based on the comprehensive consideration of the above two control indicators. r , as shown below.

[0087] f r =α1f TTS-merge +α2f TTS

[0088] In the formula, f r is the comprehensive reward function; α1 and α2 are the preset weight coefficients.

[0089] Although safety rules have been taken into consideration in the action space design, when the action space is designed as a speed change value, when the speed limit difference between adjacent lanes has reached 20km / h, the action output by the intelligent agent may cause the speed limit difference to exceed 20km / h. The output action may even result in dangerous scenarios where the lane speed limit is greater than 120km / h or less than 40km / h, which exceeds the prescribed speed limit. It is difficult to ensure the safety of the vehicle's driving process.

[0090] To avoid this situation, during the reinforcement learning training process, this study combined a set of 125 speed limits that can ensure that the lane speed limit values ​​do not differ by more than 20 km / h and meet the speed limit regulations, and designed a safety detection mechanism to verify the actions output by the agent; the safety detection mechanism process is as follows Figure 6As shown; the current speed limit value is obtained, and the intelligent transportation system outputs the speed limit change value while satisfying the safety constraints, automatically calculates a new round of speed limit values, and determines whether it is feasible. If not, repeat this step. If it is feasible, automatically calculate the next round of speed limit values.

[0091] Under the action of the safety detection mechanism, first, the speed limit value is calculated for the speed limit change value action output by the agent according to the current speed limit value, and then 125 possible speed limit value combinations are traversed. If the current speed limit value combination is not found in the possible speed limit value combination set, then the action output by the agent is regarded as an invalid action and the current speed limit value combination is maintained; otherwise, the action output by the agent is executed, and the speed limit value of each lane is calculated according to the action, as shown in the following formula.

[0092]

[0093] Where a i,t-1 ——Speed ​​limit adjustment value of the i-th lane;

[0094] ——A set of speed limit values ​​calculated based on the speed limit change value action output by the agent.

[0095] The variable speed limit control method proposed in this study is actually applied to the high-speed merging area, and combined with various traffic facilities such as detectors on the road section, real-time traffic status monitoring and the release of variable speed limit information can be carried out to form a variable speed limit control application strategy. The lane-level variable speed limit control strategy process based on reinforcement learning PER-DDQN is as follows: Figure 7 The specific implementation steps are as follows:

[0096] Step 1. Set the corresponding parameters, including the maximum number of simulation rounds N, the simulation step size T per round, the soft update rate τ, the ε decay rate k0 per round, and the minimum ε value ε min Initialize the PER-DDQN agent local Q network parameters θ and target Q network parameters

[0097] Let n = 0, t = 0;

[0098] Step 2, number of simulation rounds n = n + 1;

[0099] Step 3, simulation time step t = t + 1;

[0100] Step 4: At the tth time step, the SUMO simulation software converts the state information S collected by the sensors on the road into t Passed to the PER-DDQN agent;

[0101] Step 5: The agent selects corresponding actions using the ε-greedy strategy, that is, randomly selects an action from the action space with probability ε, or outputs an action according to the state with probability 1 - ε.

[0102] Step 6: Use the safety detection mechanism for specification, convert the output action into a speed limit value and transmit it to the vehicles in the speed limit area, and then enter the next control cycle. The agent obtains a new state S. t+1 and the reward r t ;

[0103] Step 7: Store the experience trajectory (S t , a t , S t+1 , r t ) into the experience replay pool, use the prioritized experience replay to extract samples, update the local Q-network parameters θ using gradient descent according to Equation (9), and use soft update to update the target network parameters

[0104] Step 8: If t < T, return to Step 3; otherwise, enter Step 9.

[0105] Step 9: ε = min(ε * k0, ε min ), if n < N, return to Step 2; otherwise, enter Step 10.

[0106] Step 10: The process ends.

[0107] This application first constructs a lane-level dynamic variable speed limit (DVSL) control model based on reinforcement learning, including action space optimization. In view of the difficulty in solving the high-dimensional action space faced by the DVSL control problem, it innovatively uses the speed limit change value to optimize the action space design, making the problem-solving more efficient.

[0108] State space and reward function design, determine the state space including the whole journey time and the confluence area journey time, and comprehensively consider these two factors to design the reward function, guiding the agent to learn the corresponding control strategy to ensure the effectiveness of speed limit control.

[0109] Application of DDQN algorithm and PER technology. During the model solving process, use the DDQN (double deep Q-Network) algorithm that performs well in the discrete action space, and combine it with the prioritized experience replay technology (PER), which significantly improves the training efficiency and model performance, and enhances the applicability of the model in practical applications;

[0110] Secondly, safety constraints and intelligent control are carried out. Including speed limit specifications, according to my country's highway speed limit specifications, and taking into account the impact of the speed change between adjacent lanes in the same period and between two controls in the same period on driving safety, the speed limit value output by the intelligent control system is defined to meet the following safety constraints: the speed change between adjacent lanes in the same period is not higher than 20km / h; the speed change between adjacent lanes in the same period is not higher than 10km / h.

[0111] Lane-to-lane safety detection mechanism: We explored and proposed a lane-to-lane safety detection mechanism to assist model training and ensure the feasibility and safety of the lane-level variable speed limit model.

[0112] Finally, real-time dynamic control under the road-police linkage mode, through the road-police linkage mode, combined with real-time monitoring and dynamic adjustment mechanisms, ensures that the variable speed limit control method in the merging area of ​​the highway can quickly respond to changes in traffic conditions and provide immediate traffic guidance and management.

[0113] This application significantly improves the traffic efficiency and safety of highway merging areas in terms of variable speed limit control in merging areas. Through the DVSL control model based on reinforcement learning, optimized action space design, and perfect safety constraints and lane-to-lane safety detection mechanism, it provides a new technical path and solution for the development of intelligent transportation systems.

[0114] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A variable speed limit control method for a merging area under a highway road police linkage mode, characterized in that: The method specifically includes: First, a lane-level variable speed limit control model based on reinforcement learning is constructed as follows: St1, set the corresponding parameters, including the maximum number of simulation rounds N, the simulation step size T per round, the soft update rate τ, the ε decay rate k0 per round, and the minimum ε value ε min ; Initialize the PER-DDQN agent local Q network parameters θ and target Q network parameters Let n = 0, t = 0; St2, number of simulation rounds n = n + 1; St3, simulation time step t=t+1; St4, at the t time step, the SUMO simulation software converts the state information S collected by the sensors on the road into t Passed to the PER-DDQN agent; St5, the agent uses the ε-greedy strategy to select the corresponding action, that is, randomly select an action from the action space with a probability of ε, or output an action according to the state with a probability of 1-ε St6, use the safety detection mechanism to regulate, and convert the output action into a speed limit value and pass it to the speed limit area vehicle, then enter the next control cycle, and the agent obtains a new state S t+1 And reward r t ; St7, the experience track (S t , a t , S t+1 , r t ) is stored in the experience replay pool, and the samples extracted by priority experience replay are used to update the local Q network parameters θ using gradient descent according to formula (9), and soft update is used Update target network parameters St8, if t<T, return to St3, otherwise go to St9; St9,ε=min(ε*k0,ε min ), if n<N, return to St2, otherwise go to St10; St10, the process ends; Secondly, safety constraints and intelligent control are performed to ensure that the output speed limit value satisfies the requirement that the speed change of adjacent lanes during the same period does not exceed the preset speed 1, and the speed change of adjacent lanes during the same period does not exceed the preset speed 2; Finally, human assistance is introduced. Under the combination of human assistance with real-time monitoring and dynamic adjustment mechanism, it is ensured that the variable speed limit control method in the merging area of ​​the highway can quickly respond to changes in traffic conditions and provide immediate traffic guidance and management.

2. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: The preset speed 1 is 20km / h, and the preset speed 2 is 10km / h.

3. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: Before building the lane-level variable speed limit control model, the state space needs to be defined as follows: The state space is defined to contain 19 states, as follows: In the formula, S is the state set, R up is the occupancy rate of each upstream lane, R down is the occupancy rate of each downstream lane, R bottleneckup is the occupancy rate of each lane upstream of the bottleneck area, R ramp is the occupancy rate of the upstream lane of the ramp, is the average speed of each lane upstream of the bottleneck area, is the average speed of each lane downstream of the bottleneck area, V limit It is the set of speed limit values ​​for each lane in the control area; the traffic flow information in the above-mentioned merging area is collected by detectors set at corresponding positions.

4. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: After defining the state space, we need to use the total travel time and the merging area travel time to construct a reward function; specifically: f r =α1f TTS-merge +α2f TTS ; In the formula, f r is the comprehensive reward function; α1 and α2 are the preset weight coefficients, f TTS-merge Construct a reward function for the merging area travel time; f TTS is the reward function related to the total travel time.

5. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: f TTS-merge Specifically: In the formula, f TTS-merge is the reward function related to the travel time in the merging area, k is the vehicle number passing through the merging area, and K is the number of vehicles passing through the merging area in one time step.

6. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: f TTS Determined by: First, the relationship between the inflow and outflow of the road network and the vehicle's full travel time is constructed as follows: Where Z is the total time compensation, T is the time interval, and F in F is all vehicles flowing into the road network. out All vehicles leaving the road network; because and Therefore, the reward function for the total travel time is designed as follows: f TTS =f out -f in ; In the formula, f TTS is the reward function related to the total travel time, f in is the number of vehicles flowing into the road network within a time step; f out is the number of vehicles flowing out of the road network in one time step.

7. The variable speed limit control method for a merging area in a highway road police linkage mode according to claim 1 is characterized in that: The specific methods of safety constraints and intelligent control are as follows: First, the output speed limit value must meet the following safety constraints: the speed change of adjacent lanes in the same period is not higher than 20km / h; the speed change of adjacent lanes in the same period is not higher than 10km / h; Introduce a lane-to-lane safety detection mechanism, specifically: After obtaining the current speed limit value, the intelligent transportation system outputs the speed limit change value while meeting the safety constraints, automatically calculates the new round of speed limit values, and determines whether it is feasible. If not, repeat this step. If it is feasible, automatically calculate the next round of speed limit values.

Images