A secondary accident variable speed limit prevention and control method based on deep reinforcement learning

Abstract

The present invention relates to the field of road traffic safety, and in particular to a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, which is used to reduce the probability of secondary accidents. The secondary accident variable speed limit prevention and control method based on deep reinforcement learning uses a coil detector to collect information, which is the traffic status upstream and downstream of the accident; the central speed limit controller receives and executes the intelligent networked vehicle traffic flow with a variable speed limit strategy; the method is: the coil detectors are evenly arranged at certain intervals along each lane of the highway, and return the traffic status to the central speed limit controller at a measurement interval of 30 seconds; when a traffic accident occurs, the secondary accident variable speed limit prevention and control system is triggered to initialize; the central speed limit controller issues a speed limit command, and the speed limit command is updated every 2 minutes; based on the DQN algorithm, according to the current traffic status s at time t returned by the coil detector t , the evaluation network calculates the Q value of each action in the current speed limit control action space, that is, the current Q value, and selects the action with the largest current Q value as the optimal speed limit action a t ; The speed limit command is issued to the intelligent connected vehicle in the environment, and the reward value r is obtained after the speed limit command is executed t ; Update the traffic status to s t+1 ; Use the target network to calculate the next traffic state s t+1 The target Q value of all actions is calculated; the parameters of the evaluation network are updated to determine whether the variable speed limit control is ended according to the accident situation.

Description

Technical Field

[0001] The present invention relates to the field of road traffic safety, and in particular to a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, which is used to reduce the probability of secondary accidents. Background Art

[0002] A secondary accident is defined as a chain of accidents caused by the failure of upstream vehicles to slow down or avoid in time after a primary accident. Highways have the characteristics of high traffic density and high speed. Therefore, the risk of secondary accidents on highways is much higher than that on ordinary roads, and will further expand the scope of the accident, causing more casualties and property losses. Traditional secondary accident prevention and control methods mainly rely on on-site command by traffic police and release of accident information, but these methods have problems such as slow response speed and poor flexibility, and it is difficult to meet the needs of modern transportation systems for efficiency and intelligence.

[0003] In recent years, deep reinforcement learning technology has made significant progress in the field of intelligent decision-making and control. Deep reinforcement learning combines the feature extraction ability of deep learning and the decision optimization ability of reinforcement learning, enabling intelligent agents to learn and make decisions autonomously in complex environments. In the field of transportation, with the development of intelligent connected vehicle technology and artificial intelligence, deep reinforcement learning has been widely used in autonomous driving, vehicle networking control, and traffic signal optimization, and has achieved good application results.

[0004] Therefore, applying intelligent connected vehicles and deep reinforcement learning technology to the field of secondary accident prevention and control has important research value and practical application prospects. By constructing a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, it is possible to achieve real-time perception and prediction of road conditions, adjust the speed limit strategy according to changes in traffic flow status, thereby effectively reducing the probability of secondary accidents, improving road traffic efficiency, and alleviating traffic congestion. Summary of the invention

[0005] In order to solve the above-mentioned problems, the present invention provides a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, which can timely adjust the speed limit strategy based on changes in traffic flow conditions upstream and downstream of the accident, providing a scientific and effective method for stabilizing the traffic efficiency and improving the safety level of highways.

[0006] The present invention adopts the following technical solution:

[0007] A secondary accident variable speed limit prevention and control method based on deep reinforcement learning,

[0008] The prevention and control method uses coil detectors to collect information, which is the traffic status upstream and downstream of the accident; the central speed limit controller receives and executes the intelligent networked vehicle traffic flow with variable speed limit strategy;

[0009] The steps of this method are as follows:

[0010] (1) Coil detectors are evenly spaced along each lane of the highway and return traffic status to the central speed limit controller at a measurement interval of 30 seconds;

[0011] (2) When a traffic accident occurs, the secondary accident variable speed limit prevention and control system is triggered to initialize; the central speed limit controller issues a speed limit command, which is updated every 2 minutes;

[0012] (3) Based on the DQN algorithm, the traffic status s at the current time t returned by the coil detector t , the evaluation network calculates the Q value of each action in the current speed limit control action space, that is, the current Q value, and selects the action with the largest current Q value as the optimal speed limit action a t ;

[0013] (4) The optimal speed limit action a obtained from step (3) t , a speed limit command is issued to the intelligent connected vehicle in the environment, and the reward value r is obtained after the speed limit command is executed t ; Update the traffic status to s t+1 ;

[0014] (5) The traffic status s obtained in steps (3) and (4) t , optimal speed limit action a t , reward value r t , updated traffic status t+1 Store the experience data into the experience pool;

[0015] Randomly extract a batch of experience data from the experience pool for target network training, and use the evaluation network to calculate the current state s t The Q value of all possible actions is calculated using the target network to calculate the next traffic state s t+1 The target Q value for all actions;

[0016] (6) Update the parameters of the evaluation network so that the current Q value output by the evaluation network in step (3) is close to the target Q value obtained in step (5), and regularly copy the parameters of the evaluation network to the target network to ensure the stability of the algorithm;

[0017] (7) Determine whether the variable speed limit control has ended based on the accident situation. If the accident has not ended, repeat steps (3) to (6). If the accident has ended, the variable speed limit control ends.

[0018] The secondary accident variable speed limit prevention and control method based on deep reinforcement learning described in the present invention,

[0019] The construction of a secondary accident variable speed limit prevention and control system guided by intelligent connected vehicles under mixed traffic flow includes: a central speed limit controller that solves and publishes variable speed limit strategies;

[0020] Intelligent connected vehicle traffic flow that receives and executes variable speed limit strategies;

[0021] Loop detectors that collect traffic status upstream and downstream of the accident;

[0022] The central speed limit controller and intelligent connected vehicles transmit and share information in real time based on V2X vehicle networking technology; coil detectors are evenly arranged at certain intervals along each lane of the highway, and return traffic status to the central speed limit controller at a measurement interval of 30 seconds.

[0023] The variable speed limit prevention and control method for secondary accidents based on deep reinforcement learning of the present invention, the DQN algorithm element definition in step (3) includes six elements:

[0024] Agent: The central speed limit controller is used as an agent for reinforcement learning training to solve and publish the optimal speed limit strategy

[0025] State space: The traffic flow data returned by the three consecutive sets of coil detectors closest to the upstream of the traffic accident and the two sets of coil detectors closest to the downstream are used as the state space; specifically:

[0026] The average speed {v 1 ,v 2 ,v 3}、Average flow {f 1 ,f 2 ,f 3}, average occupancy rate 1 ,o 2 ,o 3},

[0027] The average speed {v 4 ,v 5}、Average flow {f 4 ,f 5}, average occupancy rate 4 ,o 5},

[0028] The state space of the three upstream coil detectors and the two downstream coil detectors is expressed as follows:

[0029] s={v 1 ,v 2 ,v 3 ,v 4 ,v 5 ,f 1 ,f2 ,f 3 ,f 4 ,f 5 ,o 1 ,o 2 ,o 3 ,o 4 ,o 5}

[0030] Action space: Combine different speed limit ranges and speed limit values ​​in the state space to determine multiple speed limit actions to form action space a;

[0031] Strategy function: The deterministic strategy is used as the strategy function for variable speed limit control. Under a certain traffic operation state, the strategy function will select the speed limit action with the largest reward value as the output data of the central speed limit controller;

[0032] Reward value function: A comprehensive index that considers efficiency and safety is used as the reward value for solving the optimal speed limit strategy. The efficiency index uses the average speed detected by the upstream and downstream coil detectors, and the safety index uses the standard deviation of the speed detected by the upstream and downstream coil detectors. The specific calculation formula is as follows:

[0033]

[0034]

[0035] r t =α·v t -β·d t

[0036] Among them, v t is the efficiency index value at time t, d t is the safety index value at time t, r t is the reward value at time t, α and β are weight coefficients.

[0037] Evaluation network and target network: The deep neural network DNN in the DQN algorithm is used as the evaluation network and target network for variable speed limit prevention and control. The evaluation network and the target network both take the current traffic status as input and the current Q value and target Q value of all speed limit actions as output, respectively. They are the core calculation part of the central speed limit controller.

[0038] In the method for preventing and controlling secondary accidents with variable speed limits based on deep reinforcement learning described in the invention, the weight coefficients α and β of the reward value function are α=1.0 and β=0.5 respectively.

[0039] The invention describes a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, an evaluation network and a target network. These two networks have the same structure and both use a DNN neural network with two hidden layers. Each layer has 32 and 16 neurons respectively. Both use traffic status as input and the current Q value and target Q value of all possible speed limit actions as output.

[0040] In the method for preventing and controlling secondary accidents with variable speed limits based on deep reinforcement learning described in the present invention, in step (3), when iteratively calculating the Q value, the discount factor γ representing the importance of future rewards in the current decision is set to 0.9.

[0041] Beneficial Effects

[0042] The secondary accident variable speed limit prevention and control method based on deep reinforcement learning provided by the present invention can perceive the changes in the traffic environment in real time, including the location, time, traffic volume, vehicle speed, etc. of the accident, and dynamically adjust the speed limit strategy accordingly, balance traffic demand and road capacity, reduce traffic congestion and delays, and avoid high-speed collisions near the accident site, thereby significantly reducing the risk of secondary accidents and ensuring the safety of highway operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a working schematic diagram of the secondary accident variable speed limit prevention and control system of the present invention.

[0044] Figure 2 It is a schematic diagram of the layout of the coil detector of the present invention.

[0045] Figure 3 Schematic diagram of the DNN neural network of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose and technical solution of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention provides a secondary accident variable speed limit prevention and control method based on deep reinforcement learning, which can timely adjust the speed limit strategy based on changes in traffic flow conditions upstream and downstream of the accident, providing a scientific and effective method for stabilizing the traffic efficiency and improving the safety level of highways.

[0048] like Figure 1 As shown, a secondary accident variable speed limit prevention and control method based on deep reinforcement learning includes the following steps:

[0049] (1) The construction of a secondary accident variable speed limit prevention and control system guided by intelligent connected vehicles under mixed traffic flow includes three parts: a central speed limit controller that solves and publishes variable speed limit strategies, intelligent connected vehicle traffic flow that receives and executes variable speed limit strategies, and a loop detector that collects traffic status upstream and downstream of the accident. Among them, the central speed limit controller and the intelligent connected vehicle transmit and share information in real time based on V2X vehicle networking technology; the loop detectors are evenly arranged along each lane of the highway at a certain interval of L = 500m, and return the traffic status to the central speed limit controller at a measurement interval of 30 seconds.

[0050] (2) Definition of the elements of the deep reinforcement learning DQN algorithm, which includes six elements:

[0051] ① Intelligent agent: The central speed limit controller is used as an intelligent agent for reinforcement learning training to solve and publish the optimal speed limit strategy;

[0052] ② State space: The traffic flow data returned by the three consecutive sets of coil detectors closest to the upstream of the traffic accident and the two sets of coil detectors closest to the downstream are used as the state space, including the average speed {v 1 ,v 2 ,v 3}、Average flow {f 1 ,f 2 ,f 3}, average occupancy rate 1 ,o 2 ,o 3}, the average speed {v 4 ,v 5}、Average flow {f 4 ,f 5}, average occupancy rate 4 ,o 5}, state space s = {v 1 ,v 2 ,v 3 ,v 4 ,v 5 ,f 1 ,f 2 ,f 3 ,f 4 ,f 5 ,o 1 ,o 2 ,o 3 ,o 4 ,o 5};

[0053] ③Action space: Combine different speed limit ranges and speed limit values ​​to determine a variety of speed limit actions to form an action space a = {a1 ,a 2 ,…,a i}

[0054] Speed ​​limit action Speed ​​limit range R Speed ​​limit V <![CDATA[a 1 ]]> <![CDATA[R 1 ]]> <![CDATA[V 1 ]]> <![CDATA[a 2 ]]> <![CDATA[R 2 ]]> <![CDATA[V 2 ]]> … … … <![CDATA[a i ]]> <![CDATA[R i ]]> <![CDATA[V i ]]>

[0055] ④ Strategy function: The deterministic strategy is used as the strategy function for variable speed limit control. Under a certain traffic operation state, the strategy function will select the speed limit action with the largest reward value as the output of the central speed limit controller;

[0056] ⑤ Reward value function: The comprehensive index considering efficiency and safety is used as the reward value for solving the optimal speed limit strategy. The efficiency index uses the average speed detected by the upstream and downstream coil detectors, and the safety index uses the standard deviation of the speed detected by the upstream and downstream coil detectors. The specific calculation formula is as follows:

[0057]

[0058]

[0059] r t =α·v t -β·d t

[0060] Among them, v t is the efficiency index value at time t, d t is the safety index value at time t, r t is the reward value at time t, α and β are weight coefficients, α = 1.0, β = 0.5.

[0061] ⑥ Evaluation network and target network: The deep neural network (DNN) is used as the evaluation network and target network for variable speed limit control. The two networks have the same structure. Both use a DNN structure with two hidden layers, with 32 and 16 neurons in each layer respectively. Both networks take the current traffic status as input and the current Q value and target Q value of all speed limit actions as output, which are the core computing parts of the central speed limit controller.

[0062] (3) Solution of optimal speed limit control strategy based on DQN:

[0063] When a traffic accident occurs, the secondary accident variable speed limit prevention and control system is initialized and starts to issue speed limit commands. The speed limit commands are updated every 2 minutes. The discount factor γ, which represents the importance of future rewards in the current decision, is set to 0.9. According to the traffic state s at the current time t returned by the coil detector, t , DNN calculates the Q value of each action in the speed limit control action space and selects the optimal speed limit action a tThe speed limit command is issued to the intelligent connected vehicles in the environment for execution. After the action is executed, the reward value r is obtained t , the traffic status is updated to s t+1 , the experience data (state, action, reward and new state, i.e. s t ,a t ,r t ,s t+1 ) is stored in the experience pool, a batch of experience data is randomly extracted from the experience pool for network training, and the current state s is calculated using the evaluation network t The Q values ​​of all possible actions are calculated using the target network to calculate the next state s t+1 The Q values ​​of all possible actions are obtained, and the parameters of the Q network are optimized and updated so that the output of the evaluation network approaches the target Q value. The parameters of the evaluation network are regularly copied to the target network to ensure the stability of the algorithm. Finally, it is determined whether the variable speed limit control is ended. If not, the above steps are repeated.

[0064] (5) Using the above steps, the optimal strategy for variable speed limit in secondary accidents guided by intelligent connected vehicles is solved and implemented.

[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A secondary accident variable speed limit prevention and control method based on deep reinforcement learning, characterized in that : The prevention and control method uses coil detectors to collect information, which is the traffic status upstream and downstream of the accident; the central speed limit controller receives and executes the intelligent networked vehicle traffic flow with variable speed limit strategy; The steps of this method are as follows: (1) Coil detectors are evenly spaced along each lane of the highway and return traffic status to the central speed limit controller at a measurement interval of 30 seconds; (2) When a traffic accident occurs, the secondary accident variable speed limit prevention and control system is triggered to initialize; the central speed limit controller issues a speed limit command, which is updated every 2 minutes; (3) Based on the DQN algorithm, the current value returned by the coil detector Traffic status at the moment , the evaluation network calculates the Q value of each action in the current speed limit control action space, that is, the current Q value, and selects the action with the largest current Q value as the optimal speed limit action ; The definition of DQN algorithm elements includes: state space, action space, strategy function, reward value function, evaluation network and target network; State space: The traffic flow data returned by the three consecutive sets of coil detectors closest to the upstream of the traffic accident and the two sets of coil detectors closest to the downstream are used as the state space; specifically: The average speed of the three upstream coil detectors returning every 30 seconds , average flow , average share , The average speed of the two sets of coil detectors downstream every 30 seconds , average flow , average share , The state space of the three upstream coil detectors and the two downstream coil detectors is expressed as follows: Action space: Combine different speed limit ranges and speed limit values ​​in the state space to determine multiple speed limit actions to form the action space ; Strategy function: The deterministic strategy is used as the strategy function for variable speed limit control. Under a certain traffic operation state, the strategy function will select the speed limit action with the largest reward value as the output data of the central speed limit controller; Reward value function: A comprehensive index that considers efficiency and safety is used as the reward value for solving the optimal speed limit strategy. The efficiency index uses the average speed detected by the upstream and downstream coil detectors, and the safety index uses the standard deviation of the speed detected by the upstream and downstream coil detectors. The specific calculation formula is as follows: in, for Moment efficiency index value, for Safety index value at all times, for Moment reward value, and is the weight coefficient; Evaluation network and target network: The deep neural network DNN in the DQN algorithm is used as the evaluation network and target network for variable speed limit prevention and control. Both the evaluation network and the target network take the current traffic status as input and the current Q value and target Q value of all speed limit actions as output respectively; (4) The optimal speed limit action obtained from step (3) , a speed limit command is issued to the intelligent connected vehicles in the environment, and the reward value is obtained after the speed limit command is executed ; Update the traffic status to ; (5) The traffic status obtained in steps (3) and (4) , Optimal speed limit action , Reward Value , updated traffic status Store the experience data into the experience pool; Randomly extract a batch of experience data from the experience pool for target network training, and use the target network to calculate the next traffic state The target Q value for all actions; (6) Update the parameters of the evaluation network so that the current Q value output by the evaluation network in step (3) is close to the target Q value obtained in step (5), and regularly copy the parameters of the evaluation network to the target network to ensure the stability of the algorithm; (7) Determine whether the variable speed limit control has ended based on the accident situation. If the accident has not ended, repeat steps (3) to (6). If the accident has ended, the variable speed limit control ends.

2. The secondary accident variable speed limit prevention and control method based on deep reinforcement learning according to claim 1 is characterized in that : The construction of a secondary accident variable speed limit prevention and control system guided by intelligent connected vehicles under mixed traffic flow includes: a central speed limit controller that solves and publishes variable speed limit strategies; Intelligent connected vehicle traffic flow that receives and executes variable speed limit strategies; Loop detectors that collect traffic status upstream and downstream of the accident; The central speed limit controller and intelligent connected vehicles transmit and share information in real time based on V2X vehicle networking technology; coil detectors are evenly arranged at certain intervals along each lane of the highway, and return traffic status to the central speed limit controller at a measurement interval of 30 seconds.

3. According to claim 1, a secondary accident variable speed limit prevention and control method based on deep reinforcement learning is characterized by: The weight coefficient of the reward value function, , the values ​​are .

4. According to claim 1, a secondary accident variable speed limit prevention and control method based on deep reinforcement learning is characterized by: The evaluation network and the target network both use a DNN neural network with two hidden layers, each with 32 and 16 neurons respectively. Both take the traffic status as input and the current Q value and target Q value of all possible speed limit actions as output.

5. According to claim 1, a secondary accident variable speed limit prevention and control method based on deep reinforcement learning is characterized by: In step (3), when iteratively calculating the Q value, the discount factor that represents the importance of future rewards in the current decision is Set to 0.9.

Images